Method and system for remotely controlling consumer electronic device

ABSTRACT

A media system replaces content in a first sequence of media content. The media system presents the first sequence of media content to an end-user and generates a fingerprint of the sequence of media content. The fingerprint is for comparison with a plurality of reference fingerprints so as to identify the first sequence of media content and determine a reference position within the first sequence of media content. The media system sends a request for a replacement sequence of content to a content replacement system, and receives replacement media content selected based on the identified first sequence of media content. The media system presents the replacement media content to the end-user instead of the first sequence of media content. Presenting the replacement media content begins at a position in the first sequence of media content that is determined based on the reference position.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/018,050, filed Jun. 26, 2018, which is a continuation of U.S. patentapplication Ser. No. 15/614,566, filed Jun. 5, 2017, which is acontinuation of U.S. patent application Ser. No. 15/292,705, filed Oct.13, 2016, which is a continuation of U.S. patent application Serial No.15/287,116, filed Oct. 6, 2016, which application is a continuation ofU.S. patent application Ser. No. 14/659,971, filed Mar. 17, 2015, whichis a continuation of U.S. patent application Ser. No. 12/760,004, filedApr. 14, 2010, which claims the benefit of the following U.S.Provisional Patent Applications all of which are incorporated byreference herein in their entirety: U.S. Provisional Patent ApplicationNo. 61/170,586, filed Apr. 17, 2009, entitled “Method and System forRemotely Controlling Media Systems”; and U.S. Provisional PatentApplication No. 61/293,798, filed Jan. 11, 2010, entitled “Method andSystem for Remotely Controlling Consumer Electronic Devices”.

TECHNICAL FIELD

The disclosed embodiments relate generally to remotely controlling amedia system, and more particularly to directing a media system toperform specific actions, such as replacing an original sequence ofmedia content (e.g., a non-targeted advertisement in a televisionbroadcast) with replacement media content (e.g., a targetedadvertisement).

BACKGROUND

In recent years, the replacement of television advertisements withtargeted television advertisements has become an increasingly importantadvertising method. Content providers such as television networksroutinely insert generic advertisements into streamed network televisioncontent. For example, a content provider (e.g., a national cabletelevision network such as ESPN) provides a network feed that containsembedded interstitial advertisements to content distributors (e.g., acable affiliates such as Comcast, Time Warner Cable). Suchadvertisements routinely include advertisements for credit cards,national retailers or other products with broad appeal. A contentdistributor or a content provider may, want to be able to substitute areplacement advertisement for the advertisement inserted in thebroadcast feed. Merely substituting in a replacement advertisement forlocal businesses, such as a car dealership or a real estate agent doesnot provide sufficient targeting for the advertisements. While theadvertisement is local it still may have no relevancy to an individualviewer. Thus, conventional methods of providing replacement content(e.g., inserting targeted advertisements) fail to provide sufficientlysophisticated and accurate targeting of replacement media content whilemaintaining the simplicity of media systems. Rather, it would bevaluable to advertisers and thus to content providers to be able toinsert highly targeted advertisements based on other specificinformation about the likely viewer of the advertisement.

SUMMARY

The above deficiencies and other problems associated with conventionalmethods of targeting replacement content are addressed by the disclosedsystem and method for remotely controlling a consumer electronic device.In accordance with one embodiment, the distribution system disclosedherein supports content replacement through a communication network thatis distinct from the primary communication network used to provide theoriginal media content to replace television advertising content,without the need for complex replacement media content targeting systemsat the media system. Original media content (e.g., advertisementsincluded in a television network feed) is provided through acommunication network (e.g., a managed satellite television network).The original media content is overwritten with replacement media content(e.g., targeted advertisements) delivered via an alternativecommunication network (e.g., an unmanaged network such as the Internet).

In accordance with some embodiments, a media system receives a firstsequence of media content from a media content distributor over a firstcommunication network, the first sequence of media content includingassociated control data for facilitating the replacement of the firstcontent with replacement content. The media system presents the firstsequence of media content to an end-user of the media system. Whilepresenting the content to the end-user, the media system detectsassociated control data. In response to detecting the control data,sending a request to a content replacement system over a secondcommunication network that is distinct from the first communicationnetwork. The media system receives a response to the request, theresponse including the replacement media content; and presents thereplacement media content to the end-user of the media system instead ofthe first sequence of media content.

In accordance with some embodiments, the media system is a consumerelectronic device of the end-user. In some embodiments, the firstsequence of media content is a sequence of video content including videocontent requested by the end-user and advertisement video content andthe replacement media content is a sequence of video content includingone or more replacement advertisements. In some embodiments, the controldata is embedded in the original sequence of media content and concealedfrom detection. In some embodiments, the control data is concealed fromdetection using steganographic techniques.

In some embodiments, the replacement media content is selected at thecontent replacement system in accordance with predefined criteria. Insome embodiments, the first sequence of media content is anadvertisement on a television channel; the replacement media content isa targeted advertisement; and the targeted advertisement is selectedbased at least in part on demographic, psychographic or behavioralinformation of the end-user of the media system. In some embodiments,while presenting the replacement media content to the end-user, themedia system receives a request for a second sequence of media content,and in response to the request for the second sequence of media content,the media system ceases to present the replacement media content; andpresents the second sequence of media content to the end-user.

In some embodiments, while presenting the second sequence of mediacontent to the end-user, the media system continues to advance thereplacement media content, receives a request for the first sequence ofmedia content; and in response to the request for the first sequence ofmedia content, the media system performs an operation. If thereplacement media content is still advancing, the media system presentsthe replacement media content. If the replacement media content hasreached an end of the replacement media content, the media systempresents the first sequence of media content to the end-user.

In some embodiments, the predefined criteria include information aboutthe end-user of the media system. In some embodiments the informationabout the end-user of the media system is provided by at least one ofthe list comprising: a content provider, a content distributor: amanufacturer of the media system and a service provider to the end-userof the media system. In some embodiments, the predefined criteriaincludes information about a provider of the replacement media content,including at least one of the list comprising: advertiser identifier,campaign identifier. In some embodiments the control data includes oneor more of the following: an advertiser identifier, an advertisementcampaign identifier, advertisement duration, advertisement pre-rollperiod or advertisement start time, current date and time.

In some embodiments, receiving the first sequence of media contentincludes receiving one or more television channels over a managed datanetwork from a content distributor, the content distributor including atleast one of: a satellite network, radio frequency broadcast network,internet protocol television provider, and a cable network.

In some embodiments, sending the request to the content replacementsystem includes sending the request over an unmanaged data network(e.g., the Internet).

In some embodiments, receiving replacement media content includesreceiving the replacement media content as a stream over an unmanageddata network.

In some embodiments, the replacement media content has an end, and themethod further comprises, the media system detecting the end of thereplacement media content while presenting the first sequence of mediacontent; ceasing to present the replacement media content; andpresenting the first sequence of media content. In some embodiments, inresponse to presenting the replacement media content, the media systemsends usage data to the content replacement system for delivery to aprovider of the replacement media content.

In some embodiments, the method described above is implemented on amedia system comprising: one or more processors, memory, a display, andone or more programs stored in the memory, the one or more programscomprising instructions to perform the various methods described above.

In some embodiments, the method described above is implemented on amedia system comprising: a computer readable storage medium storing oneor more programs configured for execution by a computer, the one or moreprograms comprising instructions to perform the various methodsdescribed above.

In one embodiment the media system receives media content includingsteganographic control data embedded into the media content. Whilepreparing to present the media content on the media system, the mediasystem detects the steganographic control data, extracts thesteganographic control data from the content; and, in response todetecting the steganographic control data, triggers an action at themedia system based at least in part on the control data.

In accordance with some embodiments, the media system is a consumerelectronic device of the end-user. In some embodiments, triggering theaction at the media system includes sending a request to a contentreplacement system, the request including at least a subset of thecontrol data, and the method further includes: receiving replacementmedia content from the content replacement system; and presenting thereplacement media content on the media system. In some embodiments,prior to triggering the action, the media system presents the mediacontent on the media system.

In some embodiments, the presented media content is modified based atleast in part on the control data; and presenting the modified mediacontent on the media system. In some embodiments, the steganographiccontrol data is encrypted. In some embodiments, the media systemmodifies the presented content by replacing the presented content withreplacement media content. In some embodiments the triggering actionincludes presenting an interactive element in conjunction withpresenting the media content on the media system. In some embodiments,triggering the action includes beginning to run an application on themedia system.

In some embodiments, the method described above is implemented on amedia system comprising: one or more processors, memory, a display, andone or more programs stored in the memory, the one or more programscomprising instructions to perform the various methods described above.

In some embodiments, the method described above is implemented on amedia system comprising: a computer readable storage medium storing oneor more programs configured for execution by a computer, the one or moreprograms comprising instructions to perform the various methodsdescribed above.

In accordance with some embodiments, a media system receives a firstsequence of media content from a media content distributor over a firstcommunication network. The media system presents the first sequence ofmedia content to an end-user of the media system and generates afingerprint of the sequence of media content. The fingerprint is forcomparison with a plurality of reference fingerprints so as to identifythe first sequence of media content and determine a reference positionwithin the first sequence of media content. The media system sends arequest for a replacement sequence of content to a content replacementsystem over a second communication network that is distinct from thefirst communication network. The request is based at least in part onthe fingerprint. The media system receives a response to the request,the response including the replacement media content. The replacementmedia content is selected based at least in part on the identified firstsequence of media content. The media system presents the replacementmedia content to the end-user of the media system instead of the firstsequence of media content. The media system presents the replacementmedia content and begins at a position in the first sequence of mediacontent that is determined based at least in part on the referenceposition.

In accordance with some embodiments, the media system is a consumerelectronic device of the end-user. In some embodiments, the plurality ofreference fingerprints are predetermined fingerprints. In someembodiments, the first sequence of media content is a sequence of videocontent including video content requested by the end-user andadvertisement video content; and the replacement media content is asequence of video content including one or more replacementadvertisements. In some embodiments, identifying the first sequence ofmedia content includes identifying a channel and a timestamp for thefirst sequence of media content.

In some embodiments, prior to generating the fingerprint, the mediasystem receives, from the content replacement system, at least a subsetof the reference fingerprints selected from the plurality of referencefingerprints; and after generating the fingerprint, the media systemcompares the fingerprint to respective reference fingerprints of thesubset of reference fingerprints; and: if a respective referencefingerprint of the subset of reference fingerprints matches thefingerprint, the request for a replacement sequence of content includesdata indicating that the fingerprint matches the respective referencefingerprint; and if the fingerprint does not match any of the respectivereference fingerprints of the subset of reference fingerprints, therequest for a replacement sequence of content includes the fingerprint,for comparison with additional reference fingerprints at the contentreplacement system.

In some embodiments, the subset of reference fingerprints are selectedfrom the plurality of reference fingerprints based on usage data. Insome embodiments, the fingerprint is determined using luminancesampling. In some embodiments, the fingerprint is determined based onvalues sampled from multiple predefined positions in a single frame inthe sequence of media content. In some embodiments, the fingerprint isdetermined based on values sampled from a predefined position in aplurality of sequentially displayed frames in the sequence of mediacontent.

In some embodiments, the first sequence of media content is anadvertisement on a television channel; the replacement media content istargeted advertisement; and the targeted advertisement is selected basedat least in part on information selected from the list consisting of:demographic, psychographic, and behavioral information of the end-userof the media system.

In some embodiments, while presenting the replacement media content tothe end-user, the media system receives a request for a second sequenceof media content; and in response to the request for the second sequenceof media content: the media system ceases to present the replacementmedia content; and presents the second sequence of media content to theend-user. In some embodiments, while presenting the second sequence ofmedia content to the end-user: the media system continues to advance thereplacement media content; receives a request for the first sequence ofmedia content; and, in response to the request for the first sequence ofmedia content: if the replacement media content is still advancing, themedia system presents the replacement media content; and if thereplacement media content has reached an end of the replacement mediacontent, the media system presents the first sequence of media contentto the end-user.

In some embodiments, the replacement media content is selected at thecontent replacement system in accordance with predefined criteria. Insome embodiments, the predefined criteria includes information about aprovider of the replacement media content, including informationselected from the list consisting of: advertiser identifier, andcampaign identifier. In some embodiments, the predefined criteriaincludes information about the end-user of the media system. In someembodiments, the information about the end-user of the media system isprovided by an entity selected from the list consisting of: a contentprovider, a content distributor; a manufacturer of the media system, anda service provider to the end-user of the media system.

In some embodiments, receiving the first sequence of media contentincludes receiving one or more television channels over a managed datanetwork from a content distributor, the content distributor including anentity selected from the list consisting of: a satellite network, radiofrequency broadcast network, internet protocol television provider, anda cable network. In some embodiments, sending the request to the contentreplacement system includes sending the request over an unmanaged datanetwork. In some embodiments, receiving replacement media contentincludes receiving the replacement media content as a stream over anunmanaged data network.

In some embodiments, the replacement media content has an end. In theseembodiments, the device detects the end of the replacement media contentwhile presenting the replacement media content, ceases to present thereplacement media content, and presents the first sequence of mediacontent.

In some embodiments, the method described above is implemented on amedia system comprising: one or more processors, memory, a display, andone or more programs stored in the memory, the one or more programscomprising instructions to perform the various methods described above.

In some embodiments, the method described above is implemented on amedia system comprising: a computer readable storage medium storing oneor more programs configured for execution by a computer, the one or moreprograms comprising instructions to perform the various methodsdescribed above.

Thus, the system and methods described herein provide means forreplacing (e.g., accurately targeting) a first sequence of media contentwith replacement media content on media systems (e.g., consumerelectronic device) without unnecessarily increasing the complexity, sizeand cost of the media system. Additionally, this approach allows for thereal time coordination between the content providers (e.g., televisionnetworks), media systems (e.g., televisions) and the content replacementsystem (e.g., the targeted advertisement serving infrastructure) that isaccessible over the second communication network (e.g., an IP networksuch as the Internet). The use of control data in this fashion allowsthe systems core logic to be located at a content replacement system,rather than the media system, thereby reducing media systemmanufacturing cost and complexity. As the number of media systems in atelevision network far exceeds the number of content replacementsystems, this savings can be considerable. Thus, while preserving themedia systems accurately, targeted replacement sequences of mediacontent can still be delivered to the media system in real-time.

Additionally, in some embodiments the system and method described hereinprovide an improvement over conventional content replacement methodsby 1) using control data delivered with media content over a firstcommunication network to initiate activity over a second communicationnetwork, and 2) serving replacement media content (e.g., video,graphics, applications, images, text, etc.,) from the secondcommunication network solely in response to the control data receivedover the first communication network. Therefore, the disclosed systemand method simultaneously provide more valuable replacement contentwhile keeping media system manufacturing costs to a minimum by allowingstorage, logic, and processing to reside in an alternate network.

Similarly the problems of embedded control data stripping are reduced oreliminated through the use of steganographic control data. By usingsteganographic control data to indicate a media content replacementopportunity, the problem of control data stripping by third parties(e.g., a cable television company, an internet protocol televisionprovider, a satellite network, a web video provider, a mobile videoprovider, etc.) is reduced or eliminated. Thus, the use ofsteganographic control data in accordance with some embodiments ensuresthe viability and integrity of the replacement media content targetingprocess.

Additionally, in some embodiments the system and method described hereinprovide an improvement over conventional content replacement methods bygenerating fingerprint of media content received over a firstcommunication network, such as a cable television network, and comparingthe generated fingerprints with predetermined reference fingerprintsreceived over a second communication network, such as the Internet, toinitiate activity over the second communication network. Therefore, thedisclosed system and method simultaneously provide replacement content,such as targeted advertisements, that is more valuable than the mediacontent received over the first communication network, such asnon-targeted advertisements, while avoiding both: (1) problems caused byentities who remove or strip embedded data from the media content, and(2) the violation of any agreements prohibiting the transmission of datain the media content.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a distributed system for providing mediacontent to a media device in accordance with some embodiments.

FIG. 2 is a block diagram of a content provider or a content distributorin accordance with some embodiments.

FIG. 3 is a block diagram of a media system for presenting media contentin accordance with some embodiments.

FIG. 4 is a block diagram of a content replacement system for providingreplacement media content in accordance with some embodiments.

FIG. 5 is a block diagram illustrating the flow of data through adistributed system for distributing media content to a media systemusing embedded control data in accordance with some embodiments.

FIG. 6A-6E include a flow diagram of a process for remotely controllingmedia systems using embedded control data in accordance with someembodiments.

FIG. 7 is a flow diagram of a method of triggering actions usingsteganographic data in accordance with some embodiments.

FIG. 8 is a block diagram illustrating the flow of data through adistributed system for distributing media content to a media systemusing fingerprint detection in accordance with some embodiments.

FIG. 9A-9F include a flow diagram of a process for remotely controllingmedia systems using detected fingerprints in accordance with someembodiments.

Like reference numerals refer to corresponding parts throughout thedrawings.

DESCRIPTION OF EMBODIMENTS

The apparatus and methods disclosed herein supplement the ordinaryprovision of a sequence of media content (e.g., one or more televisionfeeds, streams or channels) to a media system (e.g., a television). Asused herein, a sequence of media content is any media content (e.g.,audio, video, image or text) that is dynamically presented (e.g.,changes with time) by a media system. For example, a single television(or radio) advertisement is a sequence of media content. Likewise, atelevision show with embedded interstitial advertisements is a sequenceof media content. In accordance with some embodiments, the replacementof a sequence of media content includes replacing an entire sequence ofmedia content (e.g., replacing an entire television advertisement). Inaccordance with some embodiments, the replacement of a sequence of mediacontent includes replacing a portion of a sequence of media content(e.g., replacing a single embedded interstitial advertisement in atelevision show). Similarly, “media content” or “replacement mediacontent” may be used to refer to a single sequence of media content(e.g., a single advertisement or segment of a television show that isnot interrupted by other interstitial advertisements).

The relevant functional components may be implemented using standardand/or custom computer hardware and/or software (e.g., a cable set-topbox) in combination with software, and may be provided by the contentprovider (e.g., television network) the content distributor (e.g., localnetwork affiliate) or by a contractor or vendor hired to perform therelevant function. In accordance with some embodiments a contentprovider is a computer system which assembles media content into asequence of media content (e.g., a broadcast network feed). Inaccordance with some embodiments a content distributor is a computersystem which receives the sequence of media content (optionallymodifying the sequence of media content) and distributes the sequence ofmedia content to a plurality of media systems (e.g., a cable box in acable customer's home).

Functional components used to facilitate operation of a contentprovider, content distributor, media system or content replacementsystem that are not otherwise relevant to embodiments of this inventionare omitted for clarity. As used herein, video signals includeeither: 1) both audio and a video signals, 2) video signals alone. Inaccordance with some embodiments, media content includes a transmissionthat includes only an audio signal (e.g., a radio signal). Thedistribution system disclosed herein supports the insertion of one ormore replacement sequences of media content (e.g., an advertisement or aseries of advertisements) into one or more original sequences of mediacontent (e.g., a broadcast television feed), across unmanaged networks(e.g., the Internet) without the need for complex delivery, arbitraging,or switching systems at the media system.

Attention is now directed to FIG. 1, which illustrates an overview 100of a distributed system for providing media content to a media device inaccordance with some embodiments. The Distributed System 100 includes atleast: a Content Provider 102, a Content Distributor 106, a Media System108, a Content Replacement System 110 and a Replacement Content Provider112. These components are linked together through one or morecommunication networks 104-A, 104-B, 104-C (e.g., the Internet, otherwide area networks, local area network, managed content distributionnetwork, etc.) so that the various components can communicate with eachother.

The Content Provider 102 (e.g., a television network or radio network)is any party that has access to and provides source media content (e.g.,movies, television shows, video advertisements, radio shows, music). TheContent Provider 102 creates one or more sequences of media content(e.g., television streams, feeds or channels) and provides thesesequences of media content through communication network A 104-A to theContent Distributor 106 (e.g., a network access provider). In someembodiments, communication network A 104-A is a dedicated communicationchannel (e.g., a satellite link or a fiber optic data link). In someembodiments the Content Provider 102 and the Content Distributor 106 arecombined, under the control of the same entity or otherwise closelyrelated. In this embodiment, the communication network A 104-A could beany type of data link connecting a content source such as avideo-on-demand server or locally originated programming, represented inthe Distributed System 100 as the Content Provider 102, and the ContentDistributor 106.

The Content Distributor 106 is any party that distributes sequences ofmedia content received from the Content Provider 102 throughcommunication network B 104-B to a Media System 108. In some embodimentsthe Content Distributor 106 modifies the one or more sequences of mediacontent and provides the modified content to the Media System 108. Insome embodiments, communication network B is a managed network (e.g., acable television network having cable trunks and branch nodes, aninternet protocol television network, a network of satellites and groundstations, a wide area computer network, a radio frequency broadcastnetwork with broadcast towers, a satellite distribution network, or theInternet.) As one example, the Content Distributor 106 is a cablecompany and it receives an ESPN feed from the content provider,substitutes a replacement advertisement for a local real estate agentfor an original advertisement in the ESPN feed and then distributes thetelevision channel to a set-top box in the Media System 108.

The Media System 108 is connected both to communication network B 104-Band communication network C 104-C. In accordance with some embodiments,communication network C 104-C is an unmanaged network (e.g., theInternet). In some embodiments the Media System 108 is configured tocommunicate through communication network C 104-C with a ContentReplacement System 110. In response to communications from the MediaSystem 108, the Content Replacement System 110 provides replacementsequences of media content to replace at least a portion of the sequenceof media content provided by the Content Distributor 106.

Either the Content Provider 102 or the Content Distributor 106 or bothare connected to the Content Replacement System 110 throughcommunication network C. The Content Provider 102 and/or the ContentDistributor 106 share data with the Content Replacement System 110 tocoordinate media content replacement and, in some embodiments, to shareusage data. The Content Provider 102, Content Distributor 106, and orthe Content Replacement System 110 have relationships 114 with theReplacement Content Provider 112 (e.g., targeted advertiser). TheReplacement Content Provider 112 provides the Content Replacement System110 with replacement content, and in some embodiments provides theContent Provider 102 and the Content Distributor 106 with an originalsequence of media content (e.g., an untargeted advertisement) that isreplaced with the replacement media content (e.g., a sequence of mediacontent such as a targeted advertisement). In accordance with someembodiments, Replacement Content Provider 112 includes a plurality ofdistinct content providers (e.g., advertisers or other entities whocreate and/or provide replacement media content). In accordance withsome embodiments, the Content Provider 102 and the Replacement ContentProvider 112 are under the control of the same entity or otherwiseclosely related (e.g., owned by the same corporation). In thisembodiment, the communication network C 104-C is used to communicate orotherwise transfer the Content Replacement System 110 to the ReplacementContent Provider 112.

For example, a car manufacturer pays for a television station to insertan advertisement for a minivan into an advertisement slot during atelevision show. In this example, the car manufacturer also provides atargeted advertisement for a convertible to an advertisement replacementsystem to be substituted for the minivan advertisement only whendisplaying the advertisement to television viewers between the ages of18 and 25. In this example the Replacement Content Provider 112 (e.g.,the car manufacturer) provides the original media content (e.g., minivanadvertisement) to the Content Provider 102 and the replacement content(e.g., targeted convertible advertisement) to the Content ReplacementSystem 110. When the minivan advertisement is about to be played on atelevision owned by a 20 year old, the Content Replacement System 110replaces the advertisement for the minivan with the advertisement forthe convertible car and reports the replacement to the advertiser. Insome embodiments the Replacement Content Provider 112 also has arelationship with the Content Distributor 106 similar to itsrelationship with the Content Provider 102.

It should be understood that the communication networks have beendivided merely for convenience. In one embodiment, the communicationnetwork C 104-C is distinct from communication network B 104-B, andcommunication network C 104-C is the internet and communicationnetwork-B is a managed network (e.g., a cable network). In an alternateembodiment communication network C 104-C is the same as communicationnetwork B 104-B, and the Content Replacement System 110 is located atthe Content Distributor 106. Additionally, in some embodiments theContent Replacement System 110 is combined with the Content Provider 102or the Content Distributor 106. In some embodiments, the ContentDistributor 106 is the Internet. In some embodiments, the communicationnetwork C is the Internet, a local area network (LAN), or a wide areanetwork (WAN), or a different content distributor. Embodiments of theinvention disclosed herein may use various alternative distributionnetworks known in the art. In some embodiments, various ones of thepreviously described components are connected to one or more of thecommunication networks 104 using cables such as wires, optical fibersand other transmission mediums. In other embodiments, one or more ofthese components are connected to one or more of the communicationnetworks 104 through one or more wireless networks using radio signals,satellite links, or the like.

The Media System 106 (sometimes called the “client device,” “clientcomputing device,” or “client computer”) may be any computer,television, radio, set-top box, decoder, or similar device that iscapable of receiving the sequences of media content, and preparing thesequences of media content (e.g., audio or video) for display to anend-user. Examples of suitable Media Systems 106 include, withoutlimitation, desktop computers, notebook computers, tablet computers,mobile devices such as mobile phones and personal digital assistants,direct to home satellite receivers, radios, set-top boxes, andtelevisions.

In some embodiments, the one or more Content Replacement Systems 110 isa computer server. In other embodiments the Content Replacement Systems110 include a plurality of servers such as a web interface (front endserver) one or more application servers and one or more database serverswhich are connected to each other through a network (e.g., a LAN, WAN orthe like), and which exchange information with other components of thedistributed system described above through a common interface (e.g., oneor more web servers, also called front end servers). In some otherembodiments, the front end server and the back end server are mergedinto one software application and/or hosted on one physical server.

It should be understood that, in accordance with some embodiments, theone or more Content Replacement Systems 110 may include a plurality ofdistinct content replacement systems. In some embodiments each of thedistinct content replacement systems provides content replacementservices to a distinct class of media systems (e.g., one contentreplacement system provides replacement content for mobile devices,while a second content replacement system provides replacement contentfor televisions). In some embodiments each of the distinct contentreplacement systems provides content replacement services associatedwith a distinct content provider or content distributor (e.g., onecontent replacement system provides content for ESPN and one contentreplacement system replaces content for HBO).

FIG. 2 is a block diagram illustrating a Content Provider System 102 orContent Distributor System 106 in accordance with one embodiment. TheContent Provider 102 or Content Distributor 106 typically includes oneor more processing units (e.g., CPUs) 204, one or more power sources208, one or more network or other communications interfaces 210 (e.g., acoaxial cable port, an Ethernet port, and/or a wireless port), memory206, one or more communication buses 216 for interconnecting thesecomponents, and a housing 218. The Content Provider 102 or ContentDistributor 106 optionally may include a user interface comprising adisplay device 212 and a keyboard 214. The memory 206 includeshigh-speed random access memory, such as DRAM, SRAM, DDR RAM or otherrandom access solid state memory devices; and may include non-volatilememory, such as one or more magnetic disk storage devices, optical diskstorage devices, flash memory devices, or other non-volatile solid statestorage devices. Memory 206 may optionally include one or more storagedevices remotely located from the CPU(s) 204. The memory 206, oralternately the non-volatile memory device(s) within memory 206,comprises a computer readable storage medium. In some embodiments,memory 206 or the computer readable storage medium of memory 206 storesthe following programs, modules and data structures, or any subsetthereof:

-   -   An operating system 220 that includes procedures for handling        various basic system services and for performing hardware        dependent tasks.    -   One or more network communication module(s) 222 that are used        for connecting the Content Provider 102 or the Content        Distributor 106 to other computers via the communication network        interface(s) 210 (wired or wireless) to one or more        communication networks, such as the Internet, other wide area        networks, local area networks, metropolitan area networks,        satellite links, cable networks, and so on. For a Content        Provider 102, the network communication module(s) 222 connect        the Content Provider 102 to the Content Distributor 106 through        communication network A (104-A in FIG. 1), and the Content        Replacement System 110 through communication network C (104-C in        FIG. 1). For a Content Distributor 106, the network        communication module(s) 222 connect the Content Distributor 106        to communication network A (104-A in FIG. 1), communication        network B (104-B in FIG. 1) and communication network C (104-C        in FIG. 1). It should be understood that each of the plurality        of network communication modules 222 may have a separate        communication interface (e.g., an Ethernet port, a cell phone        antenna, and a cable port), or two or more of the network        communication modules 222 may share a communication interface.    -   A master control module 224 that creates and/or modifies a        sequence of media content by combining and/or reordering media        content from one or more sources by controlling a content        streaming module 226, a content switching module 228, and/or a        control data insertion module 230.    -   One or more content streaming modules 226 for retrieving content        (e.g., movies, television shows and advertisements) from one or        more sources including remotely located content providers and        local data structures 234.    -   A content switching module 228 (optional) for switching between        a plurality of content streaming modules 226 (e.g., to switch        between a primary content stream containing a movie and a        secondary content stream containing an advertisement).    -   A fingerprint generator module 229 (optional) for generating        fingerprints of the media content for use by the Content        Replacement System 110 and the Media System 108 to identify        media content to replace.    -   A control data insertion module 230 (optional) for inserting        control data into the stream of media content (e.g., by adding        steganographic data to the stream of media content), the control        data including one or more of: an advertiser identifier, an        advertisement campaign identifier, advertisement duration,        advertisement pre-roll period or advertisement start time,        current date and time, and/or a unique identifier.    -   An encoder module 231 for encoding sequences of media content        transferred from the Content Provider 102 to the Content        Distributor 106 and encoding sequences of media content        transferred to the Media System 108 from the Content Distributor        106 and/or the Content Provider 102.    -   A multiplexer 232 (optional) for multiplexing and/or        demultiplexing a plurality of sequences of media content.    -   Data structures 233 for storing control data 234 (optional) for        use by the control data insertion module 230, media content 236        (e.g., movies, advertisements, television shows), and        fingerprint data 238 (optional) including the fingerprints        generated by the fingerprint generator module 229.

Each of the above identified programs, modules and/or data structuresmay be stored in one or more of the previously mentioned memory devices,and corresponds to a set of instructions for performing the functionsdescribed above. The above identified modules or programs (i.e., sets ofinstructions) need not be implemented as separate software programs,procedures or modules, and thus various subsets of these modules may becombined or otherwise re-arranged in various embodiments. In someembodiments, the memory 206 may store a subset of the modules and datastructures identified above. Furthermore, the memory 206 may storeadditional modules and data structures not described above. Similarly,while the above identified memory devices are shown in a singlecomputing device in a single housing, it should be understood that eachof the modules could be implemented on a separate hardware device (e.g.,the master control module could be a separate master control server andthe content streamer module could be a separate content streamingserver).

Additionally, discussion of the Content Provider 102 and the ContentDistributor 106 has been consolidated into a single section for the sakeof clarity. However, one having ordinary skill in the art would readilyunderstand that, in some embodiments, the Content Distributor 106 isdistinct from the Content Provider 102 in that the Content Provider 102creates an initial sequence of media content, which is then provideddirectly to the Content Distributor 106. The Content Distributor 106receives the sequence of media content from the Content Provider 102and, in some embodiments modifies the sequence of media content byadding or replacing advertisements and then distributes the content tothe Media System 108 (e.g., a consumer electronic device) forpresentation to an end-user of the Media System 108. Alternatively, insome embodiments, the Content Provider 102 and the Content Distributor106 are combined into a single content provider/distributor whichacquires source media content and delivers it directly to the MediaSystem 108.

FIG. 3 is a block diagram illustrating a Media System 108 (also referredto as a “client device” or “client computing device,” or “consumerelectronic device”) in accordance with one embodiment. The Media System108 typically includes one or more processing units (CPUs) 304, one ormore power sources 308, one or more network or other communicationsinterfaces 310 (e.g., a coaxial cable port, an Ethernet port, and/or awireless port), memory 306 and one or more communication buses 316 forinterconnecting these components. The Media System 108 optionally mayinclude user interface devices including a display device 312-A (forproviding visual output), a speaker device 312-B (for providing audiooutput), a wireless or wired keyboard 314, and a remote 315 fordetecting user inputs various components of the Media System 108including the display (e.g., television) and other optional hardwarecomponents 316 such as content players 317 (e.g., DVD, VHS, Blu-Ray,Digital Video Recorder) and signal receivers 318 (e.g., cable box,satellite dish, etc.). Memory 306 includes high-speed random accessmemory, such as DRAM, SRAM, DDR RAM or other random access solid statememory devices; and may include non-volatile memory, such as one or moremagnetic disk storage devices, optical disk storage devices, flashmemory devices, or other non-volatile solid state storage devices.Memory 306 may optionally include one or more storage devices remotelylocated from the CPU(s) 304. Memory 306, or alternately the non-volatilememory device(s) within the memory 306, comprises a computer readablestorage medium. In some embodiments, memory 306 or the computer readablestorage medium of the memory 306 stores one or more of the followingprograms, modules and data structures, or a subset thereof:

-   -   An operating system 320 that includes procedures for handling        various basic system services and for performing hardware        dependent tasks.    -   One or more network communication module(s) 322 that are used        for connecting the Media System 108 to other computers via the        communication network interface(s) 310 (wired or wireless) to        one or more communication networks, such as the Internet, other        wide area networks, local area networks, metropolitan area        networks, satellite links, cable networks, and so on. In        particular, the network communication module(s) 322 connect the        Media System 108 to the Content Distributor 106 through        communication network B (104-B in FIG. 1), and the Content        Replacement System 110 through communication network C (104-C in        FIG. 1). It should be understood that each of the plurality of        network communication modules 322 may have a separate        communication interface (e.g., an Ethernet port, a cell phone        antenna, and a cable port), or two or more of the network        communication modules 322 may share a communication interface.    -   One or more demultiplexer module(s) 324 for demultiplexing a        sequence of media content from a plurality of multiplexed        sequences of media content (e.g., selecting a single television        channel from a plurality of multiplexed television channels).    -   One or more content decoder modules 326 for decoding content,        including video decoders 328 (e.g., MPEG2, H.264, HDMI, DVI,        composite and analog) and audio decoders 330. In some        embodiments, the decoders act primarily as interfaces to process        component signals or serial digital uncompressed content. It        should be understood that in some embodiments, one or more of        the content decoders is implemented as a separate memory device        with a firmware decoder. In some embodiments, the Media System        108 continues to monitor the decoded content (e.g., from the        Content Provider 102 or the Content Distributor 106), for        changes that represent a change of some aspect of the decoded        content (e.g., channel, volume, signal source, or other state)        as it comes from one of the optional hardware components 316        that are feeding the signal to the Media System 108 that is        otherwise not in communication with the CPU 304 that controls        the control data detector module 332.    -   A fingerprint generator module 331 (optional) for generating        fingerprints of the media content for use in coordination with        the Content Replacement System 110 to identify media content to        replace.    -   A fingerprint matching module 333 (optional) for comparing        fingerprints received from the Content Replacement System 110 to        content fingerprints generated by the fingerprint generator        module 331.    -   A control data detector module 332 (optional) for detecting        control data inserted into the sequence of media content by the        Content Provider 102 and/or the Content Distributor 106.    -   A content selector module 334 for automatically switching        between sequences of media content from a plurality of distinct        sources (e.g., an original sequence of media content from the        Content Distributor 106 and replacement media content from the        Content Replacement System 110). In some embodiments this        switching is performed in accordance with timing information        included in the control data (e.g., by counting presentation        frames such as NTSC video presented at 30 frames/60 fields per        second). In some embodiments the timing information is based on        information about the uncompressed media content (e.g., relative        video frame distance from the first bits of a field in one of        the data headers of the sequence of media content) rather than        on information about the compressed media content (e.g., the        rate of the compressed bits delivered to the Media System 108).    -   A display formatting module 335 (optional) which converts the        sequence of media content into a standardized content        presentation format (e.g., for video media content: NTSC, ATSC,        SECAM, PAL, RGN, etc.).    -   A channel selector module (optional) 336 for selecting a        sequence of media content from a plurality of sequences of        content provided by the Content Distributor 106 to present        (e.g., the selection of a television channel by the end-user).        In some embodiments, this module is located at the media device,        while in other embodiments (as shown in FIG. 5) this module is        located remotely from the media device and is accessed by the        media device through one of the network communication module(s).    -   A unique identifier 337, (e.g., a MAC address), that is used to        uniquely identify the Media System 108 to the Content        Replacement System 110.    -   An authentication module 338 for establishing a secure        connection with the Content Replacement System 110, including a        private key or some other authentication code.    -   Local storage 339 including recorded media content 340 (e.g.,        Digital Video Recorder data and/or temporarily cached        replacement media content), user preference information 342        (e.g., the preferences of the end-user), and fingerprint data        344 (optional) including at least a subset of the fingerprints        generated by the fingerprint generator module 331 and any        fingerprints received from the Content Replacement System 110.

Each of the above identified programs, modules and/or data structuresmay be stored in one or more of the previously mentioned memory devices,and corresponds to a set of instructions for performing the functionsdescribed above. The above identified modules or programs (i.e., sets ofinstructions) need not be implemented as separate software programs,procedures or modules, and thus various subsets of these modules may becombined or otherwise re-arranged in various embodiments. In someembodiments, the memory 306 may store a subset of the modules and datastructures identified above. Furthermore, the memory 306 may storeadditional modules and data structures not described above. Similarly,while the above identified memory devices are shown in a singlecomputing device, it should be understood that each of the modules couldbe implemented on a separate hardware device (e.g., the control datadetector module 332 and/or the fingerprint generator module 331 could bephysically located in the cable set-top box, an Internet Protocoltelevision set top box, or a consumer electronics device, such as atelevision or a DVD player, Blue Ray player, Personal Video Recorder, ora personal computer used to acquire and decode television content whilethe content is presented on a separate television) without altering thefunctional relationship of the control data detector module 332 and/orthe fingerprint generator module 331 to other components in the MediaSystem 108).

FIG. 4 is a block diagram illustrating a Content Replacement System 110in accordance with one embodiment. The Content Replacement System 110typically includes one or more processing units (e.g., CPUs) 404, one ormore power sources 408, one or more network or other communicationsinterfaces 410 (e.g., a coaxial cable port, an Ethernet port, and/or awireless port), memory 406, one or more communication buses 416 forinterconnecting these components, and a housing 418. The ContentReplacement System 110 optionally may include a user interfacecomprising a display device 412 and a keyboard 414. The memory 406includes high-speed random access memory, such as DRAM, SRAM, DDR RAM orother random access solid state memory devices; and may includenon-volatile memory, such as one or more magnetic disk storage devices,optical disk storage devices, flash memory devices, or othernon-volatile solid state storage devices. Memory 406 may optionallyinclude one or more storage devices remotely located from the CPU(s)404. The memory 406, or alternately the non-volatile memory device(s)within memory 406, comprises a computer readable storage medium. In someembodiments, memory 406 or the computer readable storage medium ofmemory 406 stores the following programs, modules and data structures,or any subset thereof:

-   -   An operating system 420 that includes procedures for handling        various basic system services and for performing hardware        dependent tasks.    -   One or more network communication module(s) 422 that are used        for connecting the Content Replacement System 110 to other        computers via the communication network interface(s) 410 (wired        or wireless) to one or more communication networks, such as the        Internet, other wide area networks, local area networks,        metropolitan area networks, satellite links, cable networks, and        so on. In particular, the network communication module(s) 422        connect the Content Replacement System 110 to the Content        Provider 102, Content Distributor 106, and the Media System 108        through communication network C (104-C in FIG. 1).    -   A replacement content retrieval module 424 for retrieving        replacement content (e.g., targeted advertisements) from the        Replacement Content Providers 112 (e.g., targeted advertisers).    -   A replacement content selector 426 module for determining what        replacement media content (e.g., what targeted advertisement) to        send to the Media System 108 to replace the original sequence of        media content. In some embodiments the replacement content        selector uses content replacement parameters 428 received from        the Replacement Content Provider 112 (e.g., the company        responsible for an advertising campaign or the company whose        products are being advertised).    -   A fingerprint generator module 429 (optional) for generating        fingerprints from media content, to be used for identifying        media content to replace, in coordination with the Media System        108.    -   A fingerprint matching module 431 (optional) for matching        fingerprints received from the Content Provider 102, Content        Distributor 106 and/or generated by the fingerprint generator        module 429 with fingerprints received from the Media System 108        in order to identify media content to replace.    -   A content replacement manager 430 for managing the content        replacement process, including mediating interactions between        the various hardware and software modules described herein        (e.g., for communicating instructions to the Content Provider        102 and/or the Content Distributor 106 as to where to insert the        control data). In some embodiments the content replacement        manager 430 includes authentication services 432 (e.g.,        Kerberos) for authenticating requests for replacement media        content, and a control data generator 434 (optional) for        generating the control data based on the content replacement        parameters, as described in greater detail below.    -   A usage data collector 436 for collecting usage data from the        media device and storing the usage data in data structures.    -   A signal adjustment module (e.g., a rateshaper, transcoder, or        decoder/encoder pair) for changing the replacement content        bandwidth or other characteristics to match the transmission        parameters (e.g., the maximum bandwidth) associated with the        path in the communication network C 104-C linking the Media        System 108 with the Content Replacement System 110.    -   Data structures 438 for storing data produced by and received by        the other modules and instructions described herein, including:    -   Replacement content 440 (e.g., 441-A1, 441-A2, 441-X1, etc.)        received from the Replacement Content Provider 112. In        accordance with some embodiments, multiple versions of the same        replacement content (e.g., 441-A1, 441-A2) are stored in the        data structures 438, where each version is configured (e.g.,        compressed at a different bit rate) for transmission across a        communication network (e.g., 104-C) with different network        characteristics (e.g., lower or higher bandwidth). As one        example, Content A (version 1) 441-A1 is compressed at a low bit        rate for transmission across network connections with a low        bandwidth (e.g., dial-up internet), while Content A (version 2)        is compressed at a high bit rate for transmission across network        connections with a high bandwidth (e.g., broadband internet).    -   Usage data 442 collected by the usage data collector.    -   Control data 444 (optional) created by the content replacement        manager 430. In accordance with some embodiments, the control        data 444 includes timing information for use by the Media System        108 and/or the Content Replacement System to determine the        timing of communications (e.g., stream switching or other        actions). In some embodiments this switching is performed in        accordance with timing information included in the control data        (e.g., by counting presentation frames such as NTSC video        presented at 30 frames/60 fields per second).    -   Fingerprint data 446 (optional) including fingerprints received        from the Content Provider 102, Content Distributor 106, Media        System 108, and/or generated by the fingerprint generator module        429.

Each of the above identified programs, modules and/or data structuresmay be stored in one or more of the previously mentioned memory devices,and corresponds to a set of instructions for performing the functionsdescribed above. The above identified modules or programs (i.e., sets ofinstructions) need not be implemented as separate software programs,procedures or modules, and thus various subsets of these modules may becombined or otherwise re-arranged in various embodiments. In someembodiments, the memory 406 may store a subset of the modules and datastructures identified above. Furthermore, the memory 406 may storeadditional modules and data structures not described above. Similarly,while the above identified memory devices are shown in a singlecomputing device, it should be understood that each of the modules couldbe implemented on a separate hardware device (e.g., the contentreplacement manager could be implemented as a content replacementmanager server and the usage data collector could be implemented as aseparate usage data collection server).

Content Replacement Using Control Data

Attention is now directed towards FIG. 5, which illustrates the flow ofdata through a distributed system for distributing media content to amedia device in accordance with some embodiments (e.g., embodimentswhere media content is replaced using control data insertion anddetection). Some functional components, such as the networkcommunication interfaces, have been omitted for clarity. Media contentfor presentation on a Media System 108 (e.g., television shows fordisplay on a television) begins as source media content 236-A at theContent Provider 102. The source media content is retrieved from themedia content database 236-A by one or more content streaming modules226-A, which are controlled by the master control module 224-A to createa sequence of media content by combining several sequences of mediacontent into a single sequence of media content inserting interstitialsequences of media content into a single sequence of media content. Insome embodiments the sequence of media content is created using acontent switching module 228-A.

In accordance with some embodiments the media content from the contentstreaming module and the content switching module is uncompressed (e.g.,unencoded) media content. A control data insertion module 230-Aretrieves control data 234-A. In accordance with instructions from themaster control module 224-A, the control data insertion module 230-Ainserts control data into the sequence of media content. In someembodiments the sequence of media content with the inserted control datais encoded in an encoding module 231 into a format suitable for deliveryto the Content Distributor 106 (e.g., using an audio and/or videoencoding module to produce a profile of MPEG 2 or H.264 that iscompatible with the transmission requirements of a given ContentDistributor 106).

Take, for example, a television network creating a sequence of mediacontent for a television channel. The sequence of media content includes45 minutes of a television show and thirty half minute advertisements.The advertisements and the television show are stored in the mediacontent database 236-A and are streamed through the content streamingmodule. The master control module 224-A instructs the content streamingmodule 226 which media content to stream and (optionally) uses thecontent switching module 228 to switch between advertisements and thetelevision show. The control data insertion module 230-A inserts controldata into the television channel content indicating that one of the halfminute advertisements can be replaced by the Content Distributor 106 andthat one of the half minute advertisements can be replaced by theContent Replacement System 110 with a targeted advertisement.

The Content Distributor 106 receives the sequence of media content withthe inserted control data, and integrates the sequence of media contentinto a distribution medium such as a cable network, satellite network,IPTV network, web network, local radio frequency broadcast, mobilenetwork, etc. In some embodiments, the Content Distributor 106 decodesan encoded sequence of media content from the Content Provider 102. Whenthe Content Distributor 106 is configured to insert additional content(e.g., local advertisements), the Content Distributor 106 has a contentstreaming module 226-B for inserting media content (e.g., replacementgeographically specific advertisements) into the sequence of mediacontent, as directed by a master control module 224-B. In someembodiments, the Content Distributor 106 has a control data insertionmodule 230-B for inserting control data 234-B into the sequence of mediacontent. In some embodiments, the Content Distributor 106 encodes thesequence of media content using an encoder 231 into a format suitablefor distribution to the Media System 108. When there are a plurality ofsequences of media content, the Content Distributor 106 multiplexes aplurality of sequences of media content using multiplexer module 232before passing the sequences of media content off to the Media System108.

In accordance with some embodiments, the sequence(s) of media contentare passed off to the Media System 108 through a communication networkassociated with the Content Distributor 106 (e.g., a cable network or asatellite network). In some embodiments the sequence of media content ispassed to the Media System 108 along with a plurality of other sequencesof media content, and the plurality of sequences of media content passthrough a channel selector module 336, which selects a single sequenceof media content to deliver to the Media System 108 in accordance withinstructions received from the Media System 108. In some embodiments thechannel selector module 336 is at the Media System 108 (as shown in FIG.3) and the plurality of sequences of media content are passed to thechannel selector module in the Media System 108.

At the Media System 108 the, sequence(s) of media content are passed toa multi-format demultiplexer 324/decoder 326, which demultiplexes (ifnecessary) and decodes the selected sequence of media content (e.g.,decodes an encoded video stream from one or more of MPEG2, H.264, VC1,Quicktime, or other video format). The demultiplexed and decodedsequence of media content is passed to a control data detector whichanalyzes the content to detect any control data. If no control data isdetected, the sequence of media content is passed to the contentselector 334, which passes the sequence to output devices 312 (e.g., adisplay and/or speaker system), where the sequence of media content ispresented to the end-user.

When control data is detected, the control data detector 332 sends arequest to the replacement content manager 430 in the ContentReplacement System 110, which authenticates the request and reads thecontrol data. In some embodiments the request includes a globally uniqueidentifier of the control data detector (e.g., a globally uniqueidentifier of the Media System 108). In some embodiments, thereplacement media content has header information uniquely identifyingoriginal sequence of media content where the replacement media contentis to be inserted. The multi-format demultiplexer 324/decoder 326validates the replacement opportunity from state information maintainedby the multi-format demultiplexer 324/decoder 326, by comparing thestate information against the information contained in the header of thereplacement media content received from the Content Replacement System110.

The replacement content manager 430 passes information associated withthe control data (e.g., an advertiser identifier, an advertisementcampaign identifier, advertisement duration, advertisement pre-rollperiod or advertisement start time, current date and time) to thereplacement content selector 426, which selects appropriate replacementcontent from the replacement content database 440 based on theinformation provided by the control data detector 332 about theparameters of the sequence of media content (e.g., the length of the adto be replaced, the advertising campaign, etc.) and (optionally) dataabout the Media System 108 that is sent from the control data detector332 or stored on the Content Replacement System 110. The ContentReplacement System 110 sends replacement media content to the decoder326 in the Media System 108.

In some embodiments the replacement media content arrives at thedemultiplexer 324/decoder 326 in MPEG2 transport streams, in MP4containers, or some other type of transport. Additionally, saidreplacement media content may arrive at the demultiplexer 324/decoder326 in MPEG2, H.264, VC1, Quicktime, or other video format. In someembodiments, the replacement media content is streamed over a networkconnection (e.g., the internet) and the multi-format demultiplexer324/decoder 326 has attached storage to buffer the some portion of thebeginning of the replacement sequence of media and any relatedinformation, mitigating the effects of jitter introduced throughstreaming over the internet.

The decoder 326 decodes the replacement media content and passes thedecoded replacement media content to the content selector 334, whichswitches over from the original sequence of media content to thereplacement media content at a time that is, in accordance with someembodiments, determined by the control data (e.g., the switchsynchronizes the timing of the cutover from the chosen televisionchannel to the targeted advertisement being delivered over the Internetfrom the Content Replacement System 110). The output device 312 thenpresents the replacement media content. In some embodiments, while thesequence of media content is being presented on the output devices 312,the demultiplexer 324/decoder 326 continues to demultiplex and decodethe original sequence of media content.

After the replacement media content has ended the content selector 334automatically switches back to the original sequence of media content.In some embodiments the original sequence of media content continues toadvance while the replacement media content is presented (e.g., itrestarts at a later point in the sequence, such as by replacing oneadvertisement in the sequence with a targeted advertisement of the samelength). In some embodiments the original sequence of media content doesnot continue to advance (e.g., it restarts at the point that it was whenthe replacement media content began to play, such as by inserting anadvertisement into the middle of a movie without replacing any of thecontent of the movie, essentially pausing the movie while theadvertisement is displayed).

At various points in this process the Media System 108 provides usagedata to a usage data collector 436 in the Content Replacement System110. For example, the content selector 334 informs the usage datacollector 436 of the state changes at the content selector 334. Saidstate changes including information about the delivery of replacementmedia content (e.g., a targeted ads) to an output device 312 (e.g., atelevision). The usage data is stored in a usage data database 442 forlater use by the Content Replacement System 110. For targetedadvertisement, when the content selector 334 is located in the outputdevice (e.g., a video switch in a television) usage data is morevaluable than general “rating” data (e.g., the NIELSEN TV RATINGS of theNielsen Company) because it includes information about whether thetelevision was turned on, and thus substantially increases the accuracyof determining whether the targeted advertisement was watched.

FIGS. 6A-6E include a flowchart representing a method for remotelycontrolling a Media System 108 (e.g., a consumer electronic device),according to certain embodiments. This method may be governed byinstructions that are stored in a computer readable storage medium andthat are executed by one or more processors of one or more computersystems. Each of the operations shown in FIGS. 6A-6E may correspond toinstructions stored in a computer memory or computer readable storagemedium. The computer readable storage medium may include a magnetic oroptical disk storage device, solid state storage devices such as Flashmemory, or other non-volatile memory device or devices. The computerreadable instructions stored on the computer readable storage medium arein source code, assembly language code, object code, or otherinstruction format that is interpreted by one or more processors.

In accordance with one embodiment, the Content Provider 102 and theContent Replacement System 110 perform a series of operations (602-A)and (602-B), including in some embodiments, receiving instructions froma Replacement Content Provider 112 (e.g., receiving instructions from anadvertiser indicating the circumstances under which advertisements in atelevision broadcast stream are to be replaced with targetedadvertisements provided by the advertiser). As one example, a carmanufacturer, purchases a half minute advertising placement opportunitywith a Content Provider 102 (e.g., a television network) and pays acontent replacement service to insert a targeted advertisement in theadvertisement placement opportunity. The Content Replacement System 110obtains (603) replacement media content (e.g., a targeted advertisement)from the Replacement Content Provider (112 in FIG. 1) and obtains (604)content replacement parameters from the Replacement Content Provider 112that indicate when the replacement content should be inserted into asequence of media content. In some embodiments those parameters includethe advertiser identifier, the advertising campaign identifier, a uniqueidentifier of the device playing the advertisement and/or the startingtime of the advertisement.

The Content Provider 102 obtains (606) source media content. In someembodiments, the Content Provider 102 obtains the source media contentfrom a satellite link to a content source (e.g., a television studio).Typically, this data is processed at the Content Provider 102 into aform that lends itself to storage. Additionally, media content can bereceived from a content source by accessing the content source through aproprietary network or a public network such as the Internet. In someembodiments, a Content Distributor 106 physically delivers a storagemedium containing a digital copy of a programming content to the ContentProvider 102. In some embodiments, media content is acquired andprocessed for transmission to viewers in real time, and not first stored(e.g., a live satellite feed of a football game). In some embodimentsacquiring and processing media content for transmission to viewers inreal time includes buffering at least a portion of the content (e.g.,transmitting the media content with an eight second delay to compensatefor any variations in the rate at which the media content is beingreceived by the Content Provider 102).

In accordance with some embodiments the Content Provider 102 determines(608) one or more transition points in the content. For example, theContent Provider 102 selects three five-minute blocks for commercialbreaks in an hour-long television program. The Content ReplacementSystem 110 creates (610) control data that, in some embodiments,includes triggering keys (611) where present (e.g., when the triggeringkeys are a specific type of control data). The Content ReplacementSystem 110 sends the control data to the Content Provider 102. TheContent Provider 102 receives (612) the control data, including thetriggering keys (613). In some embodiments a triggering key is asequence of data that indicates to a computing system reading thecontrol data that substantive control data is about to be provided andshould be recorded by the computer system.

The Content Provider 102 creates a sequence of media content andincorporates (614) at least a subset of the control data and thetriggering keys into the sequence of media content. In some embodiments,incorporating the control data into the media content includes usingsteganographic techniques (e.g., digital watermarking, etc.) or othertechniques to conceal the control data in the sequence of media content,as described in greater detail below.

In some embodiments the Content Distributor 106 receives the sequence ofmedia content and identifies (616) transition points within the sequenceof media content. In some embodiments, the Content Distributor 106 alsoreceives (618) control data including (619) triggering keys. The ContentDistributor 106 may incorporate (620) additional control data into thesequence of media content and (optionally) using (621) steganographictechniques, as described in greater detail below. It should beunderstood that, in some embodiments, only the Content Provider 102incorporates control data into the sequence of media content, while inother embodiments, only the Content Distributor 106 incorporates controldata into the sequence of media content. Additionally, in someembodiments, both the Content Distributor 106 and the Content Provider102 incorporate control data into the sequence of media content. (Forexample, both a television network and a cable company have arelationship with a targeted advertiser and the television network andthe cable company each insert control data into the video feed/stream toreplace a different advertisement in the same television feed with atargeted advertisement).

The Content Distributor 106 prepares (622) the media content fordelivery to a Media System 108. In some embodiments the Media System 108sends a request indicating the selection (624) of a first sequence ofmedia content (e.g., a first television channel). In this embodiment,the Content Distributor 106 delivers (626) the first sequence of mediacontent to the Media System 108. The first sequence of media content isdecoded (628) and presented (638) by the Media System 108. For example,in a switched digital video system, the end-user requests to viewchannel “42” of a plurality of cable channels, the request goes out overthe communications network and a single channel is returned to theend-user and decoded on the end-user's home equipment (e.g., a cable boxand a television). In some embodiments the decoder reports usage data toa Content Replacement System 110, this reporting may occur over analternate communication network (e.g., a second communication network).The Content Replacement System 110 stores (630-A) the usage data, asdescribed in greater detail below.

In some embodiments, the first sequence of media content is a sequenceof video content including video content requested by the end-user andadvertisement video content. In some embodiments, the replacement mediacontent is a sequence of video content including one or more replacementadvertisements.

In some embodiments, the end-user does not send a network request for afirst sequence of media content, but rather the Content Distributor 106delivers (632) a plurality of sequences of media content (e.g.,multiplexed television channels). The Media System 108 receives theplurality of sequences of media content (e.g., multiple televisionchannels) and the end-user selects (634) a first sequence of the mediacontent. In some embodiments, the Media System 108 demultiplexes theplurality of sequences of media content and decodes (636) the selectedfirst sequence of media content. In some embodiments usage data is sentby the Media System 108 to the Content Replacement System 110 and stored(630-B) for later use.

In some embodiments, the Media System 108 receives the first sequence ofmedia content from a media Content Distributor 106 over a firstcommunication network (e.g., a managed content distribution network suchas a cable network or a satellite network), the first sequence of mediacontent includes associated control data for facilitating thereplacement of media content in the first sequence of media content withreplacement media content. The Media System 108 presents (638) the firstsequence of media content to an end-user of the Media System 108. Whilepresenting the content to the end-user, the Media System 108 detects(640) the associated control data embedded within the first sequence ofmedia content. In accordance with some embodiments, the Media System 108stores (630-C) usage data and/or sends the usage data to a ContentReplacement System 110 for later use.

In response to detecting the control data, the Media System 108 sends(641) a request to the Content Replacement System 110 over a secondcommunication network (e.g., an unmanaged network such as the Internet)that is distinct from the first communication network (e.g., a managednetwork) including at least a portion of the control data. In someembodiments the request includes an authentication message (642). Anumber of different authentication techniques may be used (e.g.,Kerberos, RSA, etc.), some of which involve multiple communicationsbetween the Media System 108 and the Content Replacement System 110. Ifthe authentication is not (646) valid, then the process ends (647), andno replacement media content is sent to the Media System 108. If theauthentication message is valid (648) and the Content Replacement System110 is able to authenticate the request, then the content replacementselects (650) replacement media content in accordance with predefinedcriteria. One having ordinary skill in the art would readily understandthat any authentication process or procedure known in the art could beused to authenticate communications between the Media System 108 and theContent Replacement System 110.

It should be noted that the media device does not request particularreplacement media content. Rather, the media device merely reports aparticular set of control data to the Content Replacement System 110,and the Content Replacement System 110 uses the communication from themedia device (including one or more of a unique identifier of the mediadevice, and the control data) along with, in some embodiments,additional data stored at the Content Replacement System 110 todetermine the best replacement media content. In this way the ContentReplacement System 110 can make a sophisticated decision as to whatreplacement media content is the most suitable for the particularend-user for the particular content replacement opportunity. Moreover,the Media System 108 is not required to do any processing other thanextracting the control data and sending it to the Content ReplacementSystem 110.

In some embodiments, parameters (e.g., predefined criteria) are providedby the Replacement Content Provider 112 (e.g., an advertiser). Theseparameters enable the Content Replacement System 110 to selectreplacement media content. In some embodiments the parameters indicatethat the replacement of an original sequence of media content withreplacement media content is authorized (e.g., by an advertiser).

As one example, the control data includes an advertisement identifierand/or an advertisement campaign identifier for the genericadvertisement to be overwritten with a targeted replacementadvertisement delivered by the Content Replacement System 110. In someembodiments, the parameters identify the specific targeted advertisementthat may be inserted as a substitute for the generic advertisement. Theadvertisement identifier and the advertisement campaign identifier areused by the Content Replacement System 110 to select an advertisementthat is authorized by the advertiser to overwrite the broadcastadvertisement based on parameters provided by the Replacement ContentProvider 112 (e.g., advertiser) and maintained in a database in theContent Replacement System 110.

Similarly, a request for replacement media content may includeparameters that enable the Content Replacement System 110 to selectadvertisements that best match the technical limitations of the MediaSystem 108. For example, such parameters may include: the video profile(e.g., SD, HD and resolution parameters) and the bandwidth (e.g., datarate) of the network connection. In a complementary implementation,these parameters inform the content replacement server to assist informatting or transcoding the substitute advertisement video so it istechnically compatible with the Media System 108.

In some embodiments, a request for replacement media content includesparameters that enable the Content Replacement System 110 to selectadvertisements that target the viewer based on program context. Theseparameters enable the Content Replacement System 110 to selectadvertisements that best match the interests of the viewer based on thecharacteristics of the content being viewed. Content characteristics canalso be referred to as content properties or content metadata. Contentmetadata that would help the Content Replacement System 110 select anadvertisement that targets a typical viewer of said content include therating of the content and/or the genre of the content (e.g., Kids,Sports, Financial News, etc.) Content metadata can be obtained in manyways. One way to obtain content metadata is by extracting the metadatafrom guide listings data. Another way of obtaining content metadata iswhere the control data includes content metadata.

A request for replacement media content may also include parameters thatenable the Content Replacement System 110 to select advertisements thattarget the viewer based on the viewer demographic, psychographic orbehavioral information. These parameters enable the Content ReplacementSystem 110 to select advertisements that best match the interests of theviewer based on the viewer's demographic information.

An example of demographic information is the Media System 108 location,such as a zip code. Media System 108 location information may be storedon the Media System 108 directly and can then be sent directly as aparameter in the request for replacement media content. In some cases,Media System 108 location information may not be stored on the MediaSystem 108 directly however it can be determined at the backend bymapping the media system identifier to the location in an accountdatabase or another database within which there is a mapping of themedia system identifier to its owner's home address. One example of sucha database is a product registration database populated at the MediaSystem 108 purchase point or populated by way of a registrationmechanism such as in the case of a warranty registration. In the casewhere media system location information is not stored directly on theMedia System 108, a parameter in the request from the Media System 108would include the identifier. The identifier would be used by a ContentReplacement System 110 component as a key into a database at theback-end to resolve the media system location information. This locationinformation would used to select an advertisement that targets thederived location.

In some embodiments, additional information about the end-user of theMedia System 108 is provided by one or more of: a Content Provider 102(e.g., a Content Provider 102 knowing what premium stations the end-userhas purchased), a Content Distributor 106 (e.g., a cable network havingdata about what channels the end-user watches); a manufacturer of theMedia System 108 (e.g., information filled out on a warrantee card); anda service provider to the end-user of the Media System 108 (e.g., a cellphone provider having information about what area codes the end-usercalls the most and what kind of cell phone the end-user owns). In someembodiments this user data (e.g., data about the end-user) may be storedin the Content Replacement System 110 and used by the ContentReplacement System 110 to select replacement media content.

The Media System 108 receives (652) the replacement content and decodes(654) the replacement media content. In accordance with someembodiments, the Content Replacement System 110 receives and stores(630-D) usage data from the decoder/multiplexer when the replacementmedia content is decoded. After receiving and decoding the replacementmedia content, the Media System 108 presents (656) the replacement mediacontent to the end-user of the Media System 108 instead of the firstsequence of media content. In some embodiments, the replacement mediacontent has an end, and the Media System 108 detects the end of thereplacement media content and ceases to present the replacement mediacontent and presents the first sequence of media content.

As one example, an end-user is watching television channel “42,” thetelevision detects that a targeted advertisement spot is coming up ineight seconds. In response to detecting the control data, the televisionrequests targeted advertisement content from the Content ReplacementSystem 110 over an internet connection. After authenticating therequest, the Content Replacement System 110 sends a targetedadvertisement, which is selected based on known demographic informationabout the end-user or the television of the end-user (e.g., providedwhen the end-user purchased the television). The replacement mediacontent is delivered to the end-user's television over the internetconnection and is played on the television instead of the regularadvertisement. When the targeted advertisement ends, the channel thatthe end-user was watching is redisplayed.

In accordance with some embodiments, receiving the first sequence ofmedia content includes receiving it from a managed television network,such as a satellite network, radio frequency broadcast network, internetprotocol television system or cable network, while accessing thereplacement media content includes accessing an unmanaged IP network,such as the Internet, or an IP network managed by another, such as in aself-contained hotel network. In a related embodiment, the method maymake use of a virtual second communication network where the secondcommunication network differs from the first communication network onone or more of layers one through six of the Open SystemsInterconnection (OSI) network layer stack. In other words, the secondcommunication network can be a virtualized separate network.

In accordance with some embodiments, while presenting the replacementmedia content to the end-user, the Media System 108 receives (658) arequest to select a second sequence of media content (e.g., to changefrom channel “42” to channel “10”). In some embodiments the Media System108 sends a request to the Content Distributor 106 for the secondsequence of media content, and the Content Distributor 106 delivers(660) sequence of media content that is, in accordance with someembodiments, being provided (661) by the Content Provider 102. The MediaSystem 108 receives (662) the second sequence of media content, andpresents (664) the second sequence of media content to the end-user. Insome embodiments, the Content Distributor 106 is sending the MediaSystem 108 a plurality of sequences of media content, and the MediaSystem 108 merely demultiplexes and decodes the requested secondsequence of media content. In either case, the Media System 108 ceasespresenting the replacement media content; and, instead presents (664)the second sequence of media content to the end-user.

Thus, even though the Media System 108 has replaced the content and isno longer displaying the first sequence of media content (e.g., thefirst channel that the end-user was watching), the Media System 108 isstill able to detect and respond to a request of the end-user to changethe sequence of media content. For example, the end-user is watchingchannel “42,” an advertisement is replaced with a targetedadvertisement. Before the targeted advertisement is finished, theend-user switches to channel “10.” In response, instead of continuing todisplay the targeted advertisement, the television switches to the newchannel (e.g., channel “10”).

In some embodiments, while presenting the second sequence of mediacontent to the end-user, the media device continues to advance thereplacement media content (e.g., the targeted advertisement continues toadvance). In some embodiments, the end-user selects the first sequenceof media content after having previously selected the second set ofmedia content, (e.g., the end-user switches back from channel “10” tochannel “42”). The media device receives a request (668) for the firstsequence of media content; and in response to the request for the firstsequence of media content, if the request was (672) made within apredefined time (e.g., the length of the replacement media content, sothat the replacement media content is still advancing), the Media System108 finishes presenting (674) the replacement media content. However, ifthe request was not (670) made within a predefined time (e.g., thelength of the replacement media content, so that the replacement mediacontent has ended), the Media System 108 resumes presenting (638) thefirst sequence of media content to the end-user.

Similarly, in another embodiment, while the replacement subset of mediacontent is being sent to the viewer display, the Media System 108continues to monitor the first sequence of media content for changes inthe characteristics of the sequence, such as changes in volume, changesin channels, invocation of Emergency Alert System (EAS), invocation ofan on-screen program guide, etc., and is responsive to such changes byeither switching back to the first sequence of media content (e.g.,television stream) from the replacement media content (e.g., targetedadvertisement stream), or makes changes in characteristics of thereplacement media content (e.g., targeted advertisement stream) receivedover the alternate distribute network to mirror changes in the monitoredfirst sequence of media content (e.g., original television stream) tothe replacement media content.

In some embodiments, the Content Replacement System 110 sends (676) thestored usage data to the Content Provider 102 and/or the ContentDistributor 106. The Content Provider 102 and the Content Distributor106 receive (678A, 678-B) the usage data and may use that data toimprove their advertising strategies. In some embodiments, the ContentReplacement System 110 stores the usage data and sells (680) the usagedata.

Steganographic Triggers

Attention is now directed toward FIG. 7, which illustrates a method oftriggering actions using steganographic data in accordance with someembodiments. FIG. 7 includes a flowchart representing a method fortriggering actions using embedded steganographic data. This method maybe governed by instructions that are stored in a computer readablestorage medium and that are executed by one or more processors of one ormore computing systems. Each of the operations shown in FIG. 7 maycorrespond to instructions stored in memory or a computer readablestorage medium. The computer readable storage medium may include amagnetic or optical disk storage device, solid state storage devicessuch as Flash memory, or other non-volatile memory device or devices.The computer readable instructions stored on the computer readablestorage medium are in source code, assembly language code, object code,or other instruction format that is interpreted by one or moreprocessors.

Steganography includes inserting a message into a plurality of data suchthat the presence of the message within the data is concealed fromdetection. In some embodiments of digital steganography, this meansincluding steganographic coding inside of a transport layer, such as adocument file, image file, program or protocol (e.g., by adjusting thechrominance of one out of every 100,000 pixels to correspond to a letterin the alphabet, a change so subtle that someone not specificallyknowing where and how to look for it is unlikely to notice it.)Steganographic data (e.g., watermarks) may be inserted into mediacontent using any method known to those skilled in the art. As oneexample, U.S. Pat. No. 6,411,725, hereby incorporated by reference inits entirety, teaches the application of watermarks in video signals tomark individual objects within a video frame with additional data.

An advantage of using steganographic triggers instead of control datathat is not steganographically encoded is that steganographicallyencoded control data is able to pass intact through most data filters,such as bandwidth shaping filters, that strip out-of-band data but leaveintact the in-band video and audio programming data (e.g., including thecontrol data that is steganographically incorporated into the audio orvideo signal). In accordance with some embodiments, using steganographyto embed the splice insert control data and associated contentreplacement opportunity (e.g., “ad avail”) information within thecontent stream itself, the potential problem of data and control datastripping by any Content Distributors 106 (cable television company,internet protocol television provider, satellite network, etc.) isreduced or eliminated.

Accordingly, a method and system is disclosed for using steganographicdata to trigger events. A Media System 108 receives (702) media contentincluding steganographic control data embedded into the media content.While preparing to present the media content on the Media System 108,the Media System 108 detects (704) the steganographic control data. Insome embodiments, preparing to present media content includesdemultiplexing and decoding media content containing the steganographiccontrol data, as discussed in greater detail above. In some embodimentspreparing to present media content includes decoding a stream ofcontent.

In some embodiments the steganographic control data is encrypted (706).In this embodiment, even if the steganographic control data is detected,the content of the steganographic message will be concealed. In someembodiments the steganographic control data is decrypted at the MediaSystem 108, while in other embodiments, at least a portion of thesteganographic control data is sent to a second computing system (e.g.,a Content Replacement System 110) without decrypting the control data.In some embodiments, the second computing system decrypts the encryptedcontrol data and sends a response to the Media System 108 (e.g.,instructions to perform an action or replacement media content). In thisembodiment, the Media System 108 is never aware of the contents of theencrypted control data.

In some embodiments the media content is presented (708) by the MediaSystem 108, prior to any action being taken with respect to thesteganographic control data. For example, while an end-user is viewing astream of video content, a control data detector in the Media System 108is concurrently looking for steganographic control data in the stream ofvideo content.

When the Media System 108 detects the steganographic control data italso extracts (710) the steganographic control data from the mediacontent. In response to detecting the steganographic control data, theMedia System 108 triggers (712) an action at the Media System 108 basedat least in part on the control data. In some embodiments the action isbased on the presence of the control data (e.g., upon detecting embeddedcontrol data, sending an encrypted portion of the control data to asecond computing device). In some embodiments, the action is based atleast in part on the content of the control data (e.g., receivingcontrol data in a television stream indicating that a replacementadvertisement should be requested and then sending a request to aContent Replacement System 110 for a replacement advertisement).

In some embodiments, triggering an action includes presenting (714) aninteractive element in conjunction with presenting the media content onthe Media System 108. For example a selectable object may appear on thedisplay of the Media System 108. In some embodiments an overlay appearson the display of the Media System 108.

In some embodiments triggering an action includes replacing (716) thepresented content with replacement media content. For example, inresponse to detecting the steganographic data, the Media System 108changes channels or displays a prerecorded video message. In someembodiments, triggering the action includes invoking (717) anapplication on the Media System 108. For example, in response todetecting steganographic control data, the Media System 108 begins torecord the current program onto a hard drive on a digital videorecorder, thus reducing the amount of time spent by the end-user tryingto set up the digital video recorder to record the television show. Inan alternative embodiment, the application is an interactive programthat works in a coordinated fashion with television programming beingwatched on a television.

In some embodiments, the triggering action at the Media System 108includes sending (718) a request to a Content Replacement System 110,the request including at least a subset of the control data (optionally,an encrypted subset of the control data), as described in greater detailabove with reference to FIGS. 6A-6F. As one example, steganographiccontrol data is detected in television broadcast feed by a Media System108 indicating a generic advertisement to replace with a targetedadvertisement; the Media System 108 extracts the steganographic controldata; the Media System 108 sends the steganographic control data to aContent Replacement System 110; the Content Replacement System 110selects a targeted replacement advertisement and sends it to the MediaSystem 108; and the Media System 108 replaces the generic advertisementwith the targeted advertisement. In some embodiments the method furtherincludes receiving (720) replacement media content from the ContentReplacement System 110, and presenting (722) the replacement mediacontent on the Media System 108. Such a system and method is discussedabove in greater detail with reference to FIGS. 6A-6F, where the controldata is steganographic control data.

In some embodiments, the Media System 108 modifies (724) the presentedmedia content based at least in part on the control data; and presents(726) the modified media content on the Media System 108. For example,upon detecting, extracting and interpreting steganographic control data,a television may switch inputs from a managed television network to anunmanaged network such as the Internet as described in greater detailabove with reference to FIGS. 6A-6F. For example, the television mayswitch inputs in order to, for example, view an advertisement or obtainaccess to view a pay-per-view channel by contacting a web site; displayan “unauthorized access” message; request additional data from thetelevision network such as a movie for later viewing, programinformation about the signal being viewed, or times the current programwill be rebroadcast; download electronic program guide (EPG) data, newsticker data, or real-time sports statistics; transmit identifyinginformation such as channel, date, and time to a viewer statisticsreporting service; or display caller identification information for avoice-over-IP telephone service.

In some embodiment, modifying the media content may include altering theappearance of the programming presentation, for example by reducing thesize of the image and optionally placing it on the screen with a varietyof other images or other reduced sized programming. Additionally, insome embodiments, the audio may be altered by augmenting or replacingthe audio with audio generated from the Media System 108, or other audioavailable from the Content Distributor 106.

Content Replacement Using Fingerprints

Attention is now directed towards FIG. 8, which illustrates the flow ofdata through a distributed system for distributing media content to amedia device in accordance with some embodiments (e.g., embodimentswhere media content is replaced using fingerprint detection andmatching). Some functional components, such as the network communicationinterfaces, have been omitted for clarity. Media content forpresentation on a Media System 108 (e.g., television shows for displayon a television) begins as source media content 236-A at the ContentProvider 102. The source media content is retrieved from the mediacontent database 236-A by one or more content streaming modules 226-A,which are controlled by the master control module 224-A to create afirst sequence of media content by combining several sequences of mediacontent into the first sequence of media content and/or insertinginterstitial sequences of media content into a single sequence of mediacontent. In some embodiments the first sequence of media content iscreated using a content switching module 228-A.

In accordance with some embodiments the media content from the contentstreaming module and the content switching module is uncompressed (e.g.,unencoded) media content. The uncompressed media content is passed to afingerprint generator module 229-A from the content streaming module226-A or the content switching module 228-A. The fingerprint generatormodule 229-A generates fingerprints based on the media content andstores fingerprint data 238-A. In some embodiments, at least a subset ofthe fingerprint data 238-A is sent to the Content Replacement System110.

In some embodiments the fingerprints are based on luminance samplingtechniques. In one embodiment, luminance sampling includes identifyingthe luminance of one or more predefined blocks in predefined positionsin a video frame. In some embodiments, a block is a subset of the pixelsin a frame (e.g., an 8×8 grid of contiguous pixels). The luminance ofthese predefined blocks are compared to the luminance of the predefinedblocks in a reference video frame, and if the relative luminance of theblocks in the video frame is within a predefined threshold of theluminance of corresponding blocks in the reference video frame, thevideo frame fingerprints match. In other embodiments, the luminancesampling includes determining the change in the luminance of one or morevisual block in predefined positions over a sequence of video frames.Similarly, for this form of luminance fingerprinting, when the changesin luminance of the blocks in the sequence of frames match the changesin luminance in corresponding blocks in a reference sequence of frameswithin a predefined threshold, the video fingerprints match. While theforegoing embodiments have been described particularly with respect toluminance sampling fingerprinting, it should be understood that anyother video fingerprinting technique could be used without departingfrom the presently claimed invention.

In some embodiments, after the fingerprints have been generated, or(optionally) while the fingerprints are concurrently being generated,the sequence of media content is encoded in an encoding module 231 intoa format suitable for delivery to the Content Distributor 106 (e.g.,using an audio and/or video encoding module to produce a profile of MPEG2 or H.264 that is compatible with the transmission requirements of agiven Content Distributor 106).

Take, for example, a television network creating a sequence of mediacontent for a television channel. The sequence of media content includes45 minutes of a television show and thirty half minute advertisements.The advertisements and the television show are stored in the mediacontent database 236-A and are streamed through the content streamingmodule. The master control module 224-A instructs the content streamingmodule 226 which media content to stream and (optionally) uses thecontent switching module 228 to switch between advertisements and thetelevision show. After the media content has been assembled into asequence of media content, but before the content has been encoded,fingerprints of the unencoded media content are created (e.g., atpredefined intervals such as every 5 seconds).

The Content Distributor 106 receives the sequence of media content fromthe Content Provider 102 and integrates the sequence of media contentinto a distribution medium such as a cable network, satellite network,IPTV network, web network, local radio frequency broadcast, mobilenetwork, etc. In some embodiments, the Content Distributor 106 decodesan encoded sequence of media content from the Content Provider 102. Whenthe Content Distributor 106 is configured to insert additional content(e.g., local advertisements), the Content Distributor 106 has a contentstreaming module 226-B for inserting media content 236-B (e.g.,replacement geographically specific advertisements) into the sequence ofmedia content, as directed by a master control module 224-B. In someembodiments, the Content Distributor 106 also has a fingerprintgenerator module 229-B for generating fingerprints from sequence ofmedia content and stores fingerprint data 238-A. In some embodiments, atleast a subset of the fingerprint data 238-B is sent to the ContentReplacement System 110. In some embodiments, after adding anyreplacement media content to the sequence of media content the ContentDistributor 106 encodes the sequence of media content using an encoder231 into a format suitable for distribution to the Media System 108.When there are a plurality of sequences of media content, the ContentDistributor 106 multiplexes a plurality of sequences of media contentusing multiplexer module 232 before passing the sequences of mediacontent off to the Media System 108.

In accordance with some embodiments, the sequence(s) of media content ispassed off to the Media System 108 through a communication networkassociated with the Content Distributor 106 (e.g., a cable network or asatellite network). In some embodiments the sequence of media content ispassed to the Media System 108 along with a plurality of other sequencesof media content, and the plurality of sequences of media content passthrough a channel selector module 336, which selects a single sequenceof media content to deliver to the Media System 108 in accordance withinstructions received from the Media System 108. In some embodiments thechannel selector module 336 is at the Media System 108 (as shown in FIG.3) and the plurality of sequences of media content are passed to thechannel selector module in the Media System 108.

At the Media System 108 the, sequence(s) of media content are passed toa multi-format demultiplexer 324/decoder 326, which demultiplexes (ifnecessary) and decodes the selected sequence of media content (e.g.,decodes an encoded video stream from one or more of MPEG2, H.264, VC1,Quicktime, or other video format). The demultiplexed and decodedsequence of media content is passed to a fingerprint generator module331 in the Media System 108, which analyzes the content to generatefingerprints. Once fingerprints have been generated, they are passed toa fingerprint matching module which compares the generated fingerprintswith fingerprint data 334 received from the Content Replacement System110. The fingerprint matching module communicates with the contentreplacement manager 430 in the Content Replacement System 110 to requestreplacement media content based on the generated fingerprint(s). In someembodiments, the fingerprint matching module 431 in the ContentReplacement System 110 performs additional steps to identify the mediacontent associated with the fingerprint received from the Media System108, including comparing the received fingerprint to fingerprint data446 stored in the Content Replacement System 110. In some embodimentsthe fingerprint data 446 includes only the fingerprint data receivedfrom the Content Provider 102 and the Content Distributor 106. In someembodiments the fingerprint data also includes fingerprints andassociated data generated by a fingerprint generator module 429 at theContent Replacement System 110. The process for identifying mediacontent associated with a fingerprint is discussed in greater detailbelow with reference to FIGS. 9A-9F.

Once the content associated with the received fingerprint has beenidentified, the replacement content manager 430 passes informationassociated with the identified media content (e.g., an advertiseridentifier, an advertisement campaign identifier, advertisementduration, advertisement pre-roll period or advertisement start time) tothe replacement content selector 424, which selects appropriatereplacement media content from the replacement content database 440based on the information provided by the replacement content manager 430about the parameters of the sequence of media content (e.g., the lengthof the advertisement to be replaced, the advertising campaign, etc.) and(optionally) data about the Media System 108 that is sent from the MediaSystem 108 or stored on the Content Replacement System 110. The ContentReplacement System 110 sends replacement media content to the decoder326 in the Media System 108.

In some embodiments the replacement media content arrives at thedemultiplexer 324/decoder 326 in MPEG2 transport streams, in MP4containers, or some other type of transport. Additionally, saidreplacement media content may arrive at the demultiplexer 324/decoder326 in MPEG2, H.264, VC1, Quicktime, or other video format. In someembodiments, the replacement media content is streamed over a networkconnection (e.g., the internet) and the multi-format demultiplexer324/decoder 326 has attached storage to buffer the some portion of thebeginning of the replacement sequence of media and any relatedinformation, mitigating the effects of jitter introduced throughstreaming over the internet.

The decoder 326 decodes the replacement media content and passes thedecoded replacement media content to the content selector 334, whichswitches over from the original sequence of media content to thereplacement media content at a time that is, in accordance with someembodiments, determined in accordance with the identified media contentassociated with the fingerprint (e.g., the switch synchronizes thetiming of the cutover from the chosen television channel to the targetedadvertisement being delivered over the Internet from the ContentReplacement System 110). The output device 312 then presents thereplacement media content. In some embodiments, while the sequence ofmedia content is being presented on the output devices 312, thedemultiplexer 324/decoder 326 continues to demultiplex and decode theoriginal sequence of media content.

After the replacement media content has ended the content selector 334automatically switches back to the original sequence of media content.In some embodiments the original sequence of media content continues toadvance while the replacement media content is presented (e.g., itrestarts at a later point in the sequence, such as by replacing oneadvertisement in the sequence with a targeted advertisement of the samelength). In some embodiments the original sequence of media content doesnot continue to advance (e.g., it restarts at the point that it was whenthe replacement media content began to play, such as by inserting anadvertisement into the middle of a movie without replacing any of thecontent of the movie, essentially pausing the movie while theadvertisement is displayed).

At various points in this process the Media System 108 provides usagedata to a usage data collector 436 in the Content Replacement System110. For example, the content selector 334 informs the usage datacollector 436 of the state changes at the content selector 334. Saidstate changes including information about the delivery of replacementmedia content (e.g., a targeted ads) to an output device 312 (e.g., atelevision). The usage data is stored in a usage data database 442 forlater use by the Content Replacement System 110. For targetedadvertisement, when the content selector 334 is located in the outputdevice (e.g., a video switch in a television) usage data is morevaluable than general “rating” data (e.g., “Nielsen Ratings”) because itincludes information about whether the television was turned on, andthus substantially increases the accuracy of determining whether thetargeted advertisement was watched.

Attention is now directed towards FIGS. 9A-9F, which include a flowchartrepresenting a method for remotely controlling a Media System 108 (e.g.,a consumer electronic device), according to certain embodiments. Thismethod may be governed by instructions that are stored in a computerreadable storage medium and that are executed by one or more processorsof one or more computer systems. Each of the operations shown in FIGS.9A-9F may correspond to instructions stored in a computer memory orcomputer readable storage medium. The computer readable storage mediummay include a magnetic or optical disk storage device, solid statestorage devices such as Flash memory, or other non-volatile memorydevice or devices. The computer readable instructions stored on thecomputer readable storage medium are in source code, assembly languagecode, object code, or other instruction format that is interpreted byone or more processors.

In accordance with one embodiment, the Content Provider 102 and theContent Replacement System 110 perform a series of operations (901-A)and (901-B), including in some embodiments, receiving instructions froma Replacement Content Provider 112 (e.g., receiving instructions from anadvertiser indicating the circumstances under which advertisements in atelevision broadcast stream are to be replaced with targetedadvertisements provided by the advertiser). As one example, a carmanufacturer, purchases a half minute advertising placement opportunitywith a Content Provider 102 (e.g., a television network) and pays acontent replacement service to insert a targeted advertisement in theadvertisement placement opportunity. The Content Replacement System 110obtains (902) replacement media content (e.g., a targeted advertisement)from the Replacement Content Provider (112 in FIG. 1) and obtains (903)content replacement parameters from the Replacement Content Provider 112that indicate when the replacement media content should be inserted intoa sequence of media content. In some embodiments those parametersinclude the advertiser identifier, the advertising campaign identifier,a unique identifier of the device playing the advertisement and/or thestarting time of the advertisement.

The Content Provider 102 obtains (904) source media content. In someembodiments, the Content Provider 102 obtains the source media contentfrom a satellite link to a content source (e.g., a television studio).Typically, this data is processed at the Content Provider 102 into aform that lends itself to storage. Additionally, media content can bereceived from a content source by accessing the content source through aproprietary network or a public network such as the Internet. In someembodiments, a Content Distributor 106 physically delivers a storagemedium containing a digital copy of a programming content to the ContentProvider 102. In some embodiments, media content is acquired andprocessed for transmission to viewers in real time, and not first stored(e.g., a live satellite feed of a football game). In some embodimentsacquiring and processing media content for transmission to viewers inreal time includes buffering at least a portion of the content (e.g.,transmitting the media content with an eight second delay to compensatefor any variations in the rate at which the media content is beingreceived by the Content Provider 102).

In accordance with some embodiments, the Content Provider 102 uses thesource media content to create a first sequence of media content (906),such as a television show that includes a plurality of 30-secondadvertisements. After creating the first sequence of media content, theContent Provider 102 generates (908) fingerprint data includingfingerprints of the first sequence of media content, and stores (909)the fingerprint data including the fingerprints and, optionally one ormore of a time stamp, the channel, and the time until the nextadvertisement begins in the data structures at the Content Provider 102.The process for generating a fingerprint in accordance with someembodiments is described in more detail above with reference to FIG. 8.In some embodiments, the fingerprint data is sent to the ContentReplacement System 110. The Content Provider 102 sends (910) the streamof media content to a Content Distributor 106 or, optionally, sends thestream of media content directly to a Media System 108.

The Content Distributor 106 receives (911) the first sequence of mediacontent from the Content Provider 102 and, optionally, adds (912) mediacontent to the first sequence of media content. For example, the ContentDistributor 106 may be authorized to replace a subset of theadvertisements in a television channel with local advertisements. Insome embodiments, after adding any media content to the first sequenceof media content, the Content Distributor 106 generates (914)fingerprint data including fingerprints of the first sequence of mediacontent and stores (915) the fingerprint data including the fingerprintsand, optionally, one or more of a time stamp, the channel, and the timeuntil the next advertisement begins in the data structures at theContent Distributor 106. The process for generating a fingerprint inaccordance with some embodiments is described in more detail above withreference to FIG. 8.

It should be understood that, in some embodiments, only the ContentProvider 102 generates fingerprints of the first sequence of mediacontent, while in other embodiments, only the Content Distributor 106generates fingerprints of the first sequence of media content.Additionally, in some embodiments, both the Content Distributor 106 andthe Content Provider 102 generate fingerprints of the first sequence ofmedia content. (For example, both a television network and a cablecompany have a relationship with a targeted advertiser and thetelevision network and the cable company each generate fingerprints ofthe video feed/stream so as to enable particular positions within thesequence of media content to be identified so that both the ContentProvider 102 and the content distributor are able to identify one ormore advertisements in the same television feed for replacement with atargeted advertisement).

In some embodiments, the fingerprint data generated by the ContentProvider 102 and the Content Distributor 106 are sent to the ContentReplacement System 110, which stores (916) the fingerprint data. In someembodiments, the Content Replacement System 110 generates (917) its ownfingerprints and stores the fingerprints and, optionally, one or more ofa time stamp, the channel, and the time until the next advertisementbegins in the data structures at the Content Distributor 106.

It should be understood that typically the Content Provider 102 and theContent Distributor 106 generate the fingerprint data, because theContent Distributor 106 and the Content Provider 102 have access to thefirst sequence of content before it is broadcast to the Media System 108and thus can generate the fingerprint data so that it can be used by theMedia System 108 to identify replaceable media content in the firstsequence of media content and insert the replacement media content.However, when the Content Replacement System 110 has access to thecontent before it is displayed to the end-user (e.g., if the ContentReplacement System 110 coordinates with the Media System 108 tointroduce a 5 minute delay in the presentation of the first sequence ofmedia content after it is received by both the Media System 108 and theContent Replacement System 110), then the Content Replacement System 110can generate the fingerprint data that is used by the Media System 108,as described in greater detail below.

In some embodiments, fingerprint data is sent (918) to a Media System108 from the Content Replacement System 110. In some embodiments, thefingerprint data includes all of the fingerprint data stored at theContent Replacement System 110. In some embodiments, the ContentReplacement System 110 selects (919) a subset of the data based on usagehistory of the Media System 108. For example, if the Content ReplacementSystem 110 has fingerprint data associated with one hundred televisionchannels, and usage data for a particular Media System 108 indicatesthat the particular Media System 108 is only regularly used to watch tenof those channels, then the Content Replacement System 110 selects thesubset of the fingerprint data that includes the fingerprint dataassociated with those ten channels and only sends the fingerprint dataassociated with those ten channels. In this example, by sending only asubset of the fingerprint data, the amount of data that is transmittedto the Media System 108 (and consequently the amount of bandwidth usedto transmit that data) is reduced by approximately ninety percent.Similarly, sending only a subset of the fingerprint data reduces the useof storage space at the Media System 108. The Media System 108 receives(920) the fingerprint data from the Content Replacement System 110 andstores the fingerprint data (921) in local data structures for use inidentifying sequences of media content and requesting replacement mediacontent, as described in greater detail below.

The Content Distributor 106 prepares (922) the media content fordelivery to a Media System 108. In some embodiments the Media System 108sends a request indicating the selection (924) of a first sequence ofmedia content (e.g., a first television channel). In this embodiment,the Content Distributor 106 delivers (926) the first sequence of mediacontent to the Media System 108. The first sequence of media content isdecoded (928) and presented (938) by the Media System 108. For example,in a switched digital video system, the end-user requests to viewchannel “42” of a plurality of cable channels, the request goes out overthe communications network and a single channel is returned to theend-user and decoded on the end-user's home equipment (e.g., a cable boxand a television). In some embodiments the decoder reports usage data toa Content Replacement System 110, this reporting may occur over analternate communication network (e.g., a second communication network).The Content Replacement System 110 stores (930-A) the usage data, asdescribed in greater detail below.

In some embodiments, the first sequence of media content is a sequenceof video content including video content requested by the end-user andadvertisement video content. In some embodiments, the replacement mediacontent is a sequence of video content including one or more replacementadvertisements.

In some embodiments, the end-user does not send a network request for afirst sequence of media content, but rather the Content Distributor 106delivers (932) a plurality of sequences of media content (e.g.,multiplexed television channels). The Media System 108 receives theplurality of sequences of media content (e.g., multiple televisionchannels) and the end-user selects (934) a first sequence of the mediacontent. In some embodiments, the Media System 108 demultiplexes theplurality of sequences of media content and decodes (936) the selectedfirst sequence of media content. In some embodiments usage data is sentby the Media System 108 to the Content Replacement System 110 and stored(930-B) for later use.

In some embodiments, the Media System 108 receives the first sequence ofmedia content from a media Content Distributor 106 over a firstcommunication network (e.g., a managed content distribution network suchas a cable network or a satellite network), where fingerprint data hasbeen generated the first sequence of media content in order tofacilitate the replacement of media content in the first sequence ofmedia content with replacement media content. The Media System 108presents (938) the first sequence of media content to an end-user of theMedia System 108. While presenting the first sequence of media contentto the end-user, the Media System 108 generates (939) a fingerprint ofthe first sequence of media content. In some embodiments, thefingerprint is generated before the first sequence is actually displayedto the end-user, so that the first sequence of media content can bereplaced with replacement media content before it is displayed to theend-user.

In some embodiments, the Media System 108 sends (940) a request to theContent Replacement System 110 for replacement media content includingthe fingerprint. For example, in embodiments where the ContentReplacement System 110 does not send any fingerprint data to the MediaSystem 108, the Media System 108 must send the fingerprint to theContent Replacement System 110, and then the fingerprint is compared(942) with reference fingerprints at the Content Replacement System 110.If a matching fingerprint is not found (944), then the process ends(945). If a matching fingerprint is found (946), and the first sequenceof media content is identified by the Content Replacement System 110,then the Content Replacement System 110 proceeds to select replacementmedia content, as described in greater detail below. In accordance withsome embodiments, the Media System 108 stores (930-C) usage data and/orsends the usage data to a Content Replacement System 110 for later use.

In embodiments where the network connection is asymmetrical so that theMedia System 108 has a much higher download bandwidth than uploadbandwidth, it is beneficial for the Content Replacement System 110 tosend fingerprints to the Media System 108 and have some of the matchingof fingerprints take place at the Media System 108, instead of havingthe Media System 108 repeatedly sending fingerprint data to the ContentReplacement System 110 for matching. This arrangement is beneficial,because sending the fingerprint data to the Media System 108 uses therelatively unlimited download bandwidth rather than the relativelylimited upload bandwidth of the Media System 108. Thus, in someembodiments, it is advantageous to store at least a subset of referencefingerprint data including reference finger prints at the Media System108, where the reference fingerprint data is predetermined fingerprintdata (e.g., data originally generated by the Content Replacement System110, the content provider and/or the Content Distributor 106). The MediaSystem 108 compares (948) the generated fingerprint to the referencefingerprints in the subset of reference fingerprint data received fromthe Content Replacement System 110. If a match is not found (950), thenthe Media System 108 sends (951) a request for replacement mediacontent, the request including the generated fingerprint. As anillustrative example, when the Content Replacement System 110 only sendsfingerprint data to the Media System 108 including fingerprints for theten most frequently watched television channels, if the Media System 108is currently displaying a sequence of media content for a televisionchannel that is not one of the ten most frequently watched televisionchannels, the Media System 108 will not have predetermined fingerprintdata that matches the sequence of media content and will have to sendthe generated fingerprint to the Content Replacement System 110 forcomparison with the entire set of reference fingerprints stored at theContent Replacement System 110.

In this embodiment, the fingerprint is received at the ContentReplacement System 110 and is compared (942) with additional referencefingerprints at the Content Replacement System 110. If a matchingfingerprint is not found (944), then the process ends (945). If amatching fingerprint is found (946), thereby identifying the firstsequence of media content, then the Content Replacement System 110proceeds to select replacement media content, as described in greaterdetail below.

In contrast, if a match to the generated fingerprint is found (952),thereby identifying the first sequence of media content, the MediaSystem 108 sends (953) a request for replacement media content to theContent Replacement System 110, the request including data indicatingthat the fingerprint matches a reference fingerprint in the subset ofreference fingerprint data. For example, if the first sequence of mediacontent that is being received by the Media System 108 is one of the tenmost frequently watched television channels, and the Media System 108identifies a reference fingerprint identifying the channel and thecurrent time of the channel. In this example, the request to the MediaSystem 108 includes an identifier of the channel and a timestamp of thechannel and sends a request to the Content Replacement System 110requesting any replacement media content (e.g., targeted advertisements)for the identified channel.

In some embodiments the request includes an authentication message. Anumber of different authentication techniques may be used (e.g.,Kerberos, RSA, etc.), some of which involve multiple communicationsbetween the Media System 108 and the Content Replacement System 110. Ifthe authentication is not valid, then the process ends, and noreplacement media content is sent to the Media System 108. If theauthentication message is valid and the Content Replacement System 110is able to authenticate the request, then the content replacementselects replacement media content in accordance with predefinedcriteria. An exemplary authentication process is described in greaterdetail above with reference to FIG. 6C. One having ordinary skill in theart would readily understand that any authentication process orprocedure known in the art could be used to authenticate communicationsbetween the Media System 108 and the Content Replacement System 110.

It should be noted that the media device does not request a particularreplacement media content. Rather, the media device merely reports aparticular fingerprint or fingerprint match (e.g., an identified firstsequence of media content) to the Content Replacement System 110, andthe Content Replacement System 110 uses the request from the mediadevice (including one or more of a unique identifier of the mediadevice, and fingerprint or fingerprint match) along with, in someembodiments, additional data stored at the Content Replacement System110. The Content Replacement System 110 identifies (954) the firstsequence of media content from the matched reference fingerprint (e.g.,the Content Replacement System 110 determines that the first sequence ofmedia content is a particular television channel). The ContentReplacement System 110 also uses the matched reference fingerprint todetermine (956) a reference position (e.g., a timestamp or number offrames from a content transition point) within the first sequence ofmedia content. In this way the Content Replacement System 110 can make asophisticated decision as to what replacement media content is the mostsuitable for the particular Media System 108 and for the particularcontent replacement opportunity.

The Content Replacement System 110 selects (958) replacement mediacontent in accordance with predefined criteria. In some embodiments thepredefined criteria is based (959) at least in part on the firstsequence of media content. In some embodiments the predefined criteriais based (960) at least in part on the reference position within themedia content. In some embodiments, the predefined criteria is based(961) at least in part on information about the end-user of the MediaSystem 108 such, including replacement parameters provided by theReplacement Content Provider 112 (e.g., an advertiser). These parametersenable the Content Replacement System 110 to intelligently selectreplacement media content. In some embodiments the parameters indicatethat the replacement of a portion of the first sequence of media contentwith replacement media content is authorized (e.g., that the replacementof a non-targeted advertisement with a targeted advertisement isauthorized by the advertiser).

As one example of this method, the reference fingerprint is used by theContent Replacement System 110 to identify a sequence of media content(e.g., a television channel and/or a particular advertisement ortelevision show within the television channel) and a timestamp withinthe sequence of media content (e.g., the location of the content that isto be replaced, such as a non-targeted advertisement). A particularposition (e.g., a position relative to the reference position) withinthe identified first sequence of media content is associated with anadvertisement identifier and/or an advertisement campaign identifier forthe generic advertisement to be overwritten with a targeted replacementadvertisement delivered by the Content Replacement System 110. In someembodiments, the parameters identify the specific targeted advertisementthat may be inserted as a substitute for the generic advertisement. Theadvertisement identifier and the advertisement campaign identifier areused by the Content Replacement System 110 to select an advertisementthat is authorized by the advertiser to overwrite the broadcastadvertisement based on parameters provided by the Replacement ContentProvider 112 (e.g., advertiser) and maintained in a database in theContent Replacement System 110.

Similarly, a request for replacement media content may includeparameters that enable the Content Replacement System 110 to selectadvertisements that best match the technical limitations of the MediaSystem 108. For example, such parameters may include: the video profile(e.g., SD, HD and resolution parameters) and the bandwidth (e.g., datarate) of the network connection. In a complementary implementation,these parameters inform the content replacement server to assist informatting or transcoding the substitute advertisement video so it istechnically compatible with the Media System 108.

In some embodiments, a request for replacement media content includesparameters that enable the Content Replacement System 110 to selectadvertisements that target the viewer based on program context. Theseparameters enable the Content Replacement System 110 to selectadvertisements that best match the interests of the viewer based on thecharacteristics of the content being viewed. Content characteristics canalso be referred to as content properties or content metadata. Contentmetadata that would help the Content Replacement System 110 select anadvertisement that targets a typical viewer of said content include therating of the content and/or the genre of the content (e.g., Kids,Sports, Financial News, etc.) Content metadata can be obtained in manyways. One way to obtain content metadata is by extracting the metadatafrom guide listings data.

A request for replacement media content may also include parameters thatenable the Content Replacement System 110 to select advertisements thattarget the viewer based on the viewer demographic, psychographic orbehavioral information. These parameters enable the Content ReplacementSystem 110 to select advertisements that best match the interests of theviewer based on the viewer's demographic information.

An example of demographic information is the media system location, suchas a zip code. Media System 108 location information may be stored onthe Media System 108 directly and can then be sent directly as aparameter in the request for replacement media content. In some cases,media system location information may not be stored on the Media System108 directly however it can be determined at the back-end by mapping theMedia System 108 host device identifier to the location in an accountdatabase or another database within which there is a mapping of themedia system identifier to its owner's home address. One example of sucha database is a product registration database populated at the MediaSystem 108 purchase point or populated by way of a registrationmechanism such as in the case of a warranty registration. In the casewhere media system location information is not stored directly on theMedia System 108, a parameter in the request from the Media System 108would include the identifier. The identifier would be used by a ContentReplacement System 110 component as a key into a database at theback-end to resolve the media system location information. This locationinformation would be used to select an advertisement that targets thederived location.

In some embodiments, additional information about the end-user of theMedia System 108 is provided by one or more of: a Content Provider 102(e.g., a Content Provider 102 knowing what premium stations the end-userhas purchased), a Content Distributor 106 (e.g., a cable network havingdata about what channels the end-user watches); a manufacturer of theMedia System 108 (e.g., information filled out on a warrantee card); anda service provider to the end-user of the Media System 108 (e.g., a cellphone provider having information about what area codes the end-usercalls the most and what kind of cell phone the end-user owns). In someembodiments this user data (e.g., data about the end-user) may be storedin the Content Replacement System 110 and used by the ContentReplacement System 110 to select replacement media content.

In some embodiments, before the replacement media content is used toreplace the first sequence of media content at the Media System 108, theContent Replacement System 110 verifies that the first sequence of mediacontent is the identified sequence of media content. In one embodiment,the fingerprints discussed above are weak fingerprints (i.e.,fingerprints that include a relatively small amount of information, suchas a fingerprint that is generated using two blocks of a single frame ofthe media content) that is not sufficient to uniquely identify thesequence of media content, but is sufficient to identify a likelycandidate for the first sequence of media content when correlated withother available information (e.g., the approximate time that thefingerprint was processed). In this embodiment, after the sequence ofmedia content has been preliminarily identified the Media System 108generates a stronger fingerprint (i.e. a fingerprint that includes arelatively larger amount of information, such as a fingerprint generatedusing every block in a single frame of the media content) that issufficient to uniquely identify the media sequence. This strongfingerprint is compared with a corresponding fingerprint for theidentified sequence of media content. If the strong fingerprint matches,then the match is verified, and the replacement media content is sent tothe Media System 108.

As an example of this embodiment, if the first sequence of media contentis a television channel including a television program and a pluralityof advertisements, the Media System 108 initially generates weakfingerprints, which are matched (either at the Media System 108 or atthe Content Replacement System 110) with reference fingerprints todetermine the current channel. The Content Replacement System 110identifies a targeted advertisement that is going to be displayed on thechannel as a replacement for a non-targeted advertisement. However,before the non-targeted advertisement is replaced, the Media System 108generates a strong fingerprint of the identified advertisement (e.g., afingerprint of the first frame of the advertisement) which is comparedwith a strong fingerprint of the first frame of the advertisement. Insome embodiments the strong fingerprint is globally unique and isgenerated by sampling the luminance of every block in the first frame ofthe non-targeted advertisement and is compared with a correspondingreference fingerprint for the first frame of the non-targetedadvertisement. In these embodiments, the advertisement is only replacedif the strong fingerprints match. Thus, the content is replaced only ifthe non-targeted advertisement is verified to be the replaceableadvertisement using a strong fingerprint.

In some embodiments, weak fingerprints are used to continuously monitora first sequence of media content to determine whether the firstsequence of media content has ceased to be displayed (e.g., because theend-user switched channels). In this case, a very weak fingerprint maybe used (e.g., the luminance of a single block every 10 video frames)which requires very little bandwidth to communicate with the ContentReplacement System 110, but is sufficient to determine whether thesequence of media content has been changed (e.g., because the end-userhas changed the video input from cable to a DVD player or because theend-user has changed channels.)

The Media System 108 receives (962) the replacement media content anddecodes (964) the replacement media content. In accordance with someembodiments, the Content Replacement System 110 receives and stores(930-D) usage data from the decoder/multiplexer when the replacementmedia content is decoded.

After receiving and decoding the replacement media content, the MediaSystem 108 determines (966) a position to begin presenting thereplacement media content based on the reference position. As oneexample, the content to be replaced is an advertisement, and when theMedia System 108 generates the fingerprint, the Media System 108determines a frame of the fingerprint (e.g., the first frame that wasanalyzed to generate the fingerprint) and records this as part of thefingerprint data. When the sequence of media content is identified, theContent Replacement System 110 sends the Media System 108 replacementmedia content, where the replacement media content starts at a positionrelative to the fingerprint used to identify the sequence of mediacontent (e.g., the position is “239 frames after the frame associatedwith the fingerprint”). The Media System 108 uses the stored fingerprintdata about the location of the fingerprint and the information from theContent Replacement System 110 about the relative position of thefingerprint and an insertion point for the replacement media content todetermine when to begin presenting the replacement media content at theMedia System 108.

In accordance with some embodiments, after determining when to beginpresenting the replacement media content, the Media System 108 beginspresenting (968) the replacement media content instead of the firstsequence of media content. In some embodiments, the replacement mediacontent has an end, and the Media System 108 detects the end of thereplacement media content and ceases to present the replacement mediacontent and presents the first sequence of media content. In accordancewith some embodiments, when the Media System 108 begins presenting thereplacement media content, the Content Replacement System 110 receivesand stores (930-E) usage data from the decoder/multiplexer.

As one example, an end-user is watching television channel “42,” thetelevision detects that a targeted advertisement spot is coming up ineight seconds. In response to generating a fingerprint and determining amatching reference fingerprint, thereby identifying the currentlydisplayed television channel, the television requests targetedadvertisement content from the Content Replacement System 110 over aninternet connection (or sends the fingerprint to the Content ReplacementSystem 110 so that the Content Replacement System 110 can make thisdetermination). After verifying that the sequence of media contentincludes the replaceable content, the Content Replacement System 110sends a targeted advertisement, which is selected based on knowndemographic information about the end-user or the television of theend-user (e.g., provided when the end-user purchased the television).The replacement media content is delivered to the end-user's televisionover the internet connection and is played on the television instead ofthe regular advertisement. When the targeted advertisement ends, thechannel that the end-user was watching is redisplayed.

In accordance with some embodiments, receiving the first sequence ofmedia content includes receiving it from a managed television network,such as a satellite network, radio frequency broadcast network, internetprotocol television system or cable network, while accessing thereplacement media content includes accessing an unmanaged IP network,such as the Internet, or an IP network managed by another, such as in aself-contained hotel network. In a related embodiment, the method maymake use of a virtual second communication network where the secondcommunication network differs from the first communication network onone or more of layers one through six of the Open SystemsInterconnection (OSI) network layer stack. In other words, the secondcommunication network can be a virtualized separate network.

In accordance with some embodiments, while presenting the replacementmedia content to the end-user, the Media System 108 receives (969) arequest to select a second sequence of media content (e.g., to changefrom channel “42” to channel “10”). In some embodiments the Media System108 sends a request to the Content Distributor 106 for the secondsequence of media content, and the Content Distributor 106 delivers(970) sequence of media content that is, in accordance with someembodiments, being provided (971) by the Content Provider 102. The MediaSystem 108 receives (972) the second sequence of media content, andpresents (974) the second sequence of media content to the end-user. Insome embodiments, the Content Distributor 106 is sending the MediaSystem 108 a plurality of sequences of media content, and the MediaSystem 108 merely demultiplexes and decodes the requested secondsequence of media content. In either case, the Media System 108 ceasespresenting the replacement media content; and, instead presents (974)the second sequence of media content to the end-user.

Thus, even though the Media System 108 has replaced the content and isno longer displaying the first sequence of media content (e.g., thefirst channel that the end-user was watching), the Media System 108 isstill able to detect and respond to a request of the end-user to changethe sequence of media content. For example, the end-user is watchingchannel “42,” an advertisement is replaced with a targetedadvertisement. Before the targeted advertisement is finished, theend-user switches to channel “10.” In response, instead of continuing todisplay the targeted advertisement, the television switches to the newchannel (e.g., channel “10”).

In some embodiments, while presenting the second sequence of mediacontent to the end-user, the media device continues to advance thereplacement media content (e.g., the targeted advertisement continues toadvance). In some embodiments, the end-user selects the first sequenceof media content after having previously selected the second set ofmedia content, (e.g., the end-user switches back from channel “10” tochannel “42”). The media device receives a request (978) for the firstsequence of media content; and in response to the request for the firstsequence of media content, if the request was (980) made within apredefined time (e.g., the length of the replacement media content, sothat the replacement media content is still advancing), the Media System108 finishes presenting (982) the replacement media content. However, ifthe replacement media content if the request was not (984) made within apredefined time (e.g., the length of the replacement media content, sothat the replacement media content has ended), the Media System 108resumes presenting (938) the first sequence of media content to theend-user. In accordance with some embodiments, when the Media System 108resumes presenting the replacement media content, the ContentReplacement System 110 receives and stores (930-F) usage data from thedecoder/multiplexer.

Similarly, in another embodiment, while the replacement subset of mediacontent is being sent to the viewer display, the Media System 108continues to monitor the first sequence of media content for changes inthe characteristics of the sequence, such as changes in volume, changesin channels, invocation of Emergency Alert System (EAS), invocation ofan on-screen program guide, etc., and is responsive to such changes byeither switching back to the first sequence of media content (e.g.,television stream) from the replacement media content (e.g., targetedadvertisement stream), or makes changes in characteristics of thereplacement media content (e.g., targeted advertisement stream) receivedover the alternate distribute network to mirror changes in the monitoredfirst sequence of media content (e.g., original television stream) tothe replacement media content.

In some embodiments, the Content Replacement System 110 sends (986) thestored usage data to the Content Provider 102 and/or the ContentDistributor 106. The Content Provider 102 and the Content Distributor106 receive (988-A, 988-B) the usage data and may use that data toimprove their advertising strategies. In some embodiments, the ContentReplacement System 110 stores the usage data and sells (990) the usagedata.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the disclosed system and method to the precise forms disclosed.Many modifications and variations are possible in view of the aboveteachings. The embodiments were chosen and described in order to bestexplain the principles of the disclosed system and method and itspractical applications, to thereby enable others skilled in the art tobest utilize the disclosed system and method and various embodimentswith various modifications as are suited to the particular usecontemplated.

The invention claimed is:
 1. A method comprising: receiving usage datafor a client device that is configured to present media contentsequences; based on the received usage data, selecting, from among aplurality of reference fingerprints, a subset of reference fingerprints;sending to the client device, the selected subset of referencefingerprints; receiving from the client device, an indication that theclient device has detected a match between (i) one of more of theselected subset of reference fingerprints, and (ii) one or morefingerprints generated by the client device; and determining, based onthe received indication from the client device, an identity of a mediasequence being presented on the client device.
 2. The method of claim 1,wherein the usage data indicates a state change of a content selectormodule of the client device.
 3. The method of claim 1, wherein the usagedata indicates information about one or more replacement media sequencessent to the client device.
 4. The method of claim 1, wherein the usagedata indicates that the client device is routinely tuned to a subset ofmultiple channels from among a plurality of channels, and whereinselecting the subset of reference fingerprints comprises selecting asubset of reference fingerprints that corresponds to the subset ofmultiple channels.
 5. The method of claim 1, wherein selecting thesubset of reference fingerprints comprises: selecting, based on thereceived usage data, a subset of multiple channels from among aplurality of channels, each of the plurality of channels havingcorresponding reference fingerprints, and selecting the subset ofreference fingerprints that correspond to the selected subset ofmultiple channels.
 6. The method of claim 1, wherein the client deviceis a television and the media content sequences are video contentsequences.
 7. The method of claim 1, further comprising performing therecited operations in connection with each of multiple different clientdevices and wherein a different subset of reference fingerprints is sentto each of the multiple client devices.
 8. The method of claim 1,further comprising: receiving from the client device, a request for areplacement media content sequence, wherein the request was generated bythe client device in response to the client device detecting the matchbetween (i) the one or more reference fingerprints in the subset ofreference fingerprints and (ii) the one or more fingerprints generatedby the client device; responsive to receiving the request from theclient device, selecting a replacement media content sequence; andperforming an action that facilitates causing the client device topresent the selected replacement media content sequence in place of anoriginal media content sequence.
 9. The method of claim 8, wherein therequest for the replacement media content sequence comprises the one ormore fingerprints generated by the client device.
 10. The method ofclaim 8, wherein the request for the replacement media content sequencecomprises an indication that the client device detected the match. 11.The method of claim 8, wherein the action comprises sending the selectedreplacement media content sequence to the client device.
 12. The methodof claim 8, wherein the original media content sequence is an originalcommercial and the replacement media content sequence is a replacementcommercial.
 13. A server device comprising one or more processors,wherein the server device is configured to perform operationscomprising: receiving usage data for a client device that is configuredto present media content sequences; based on the received usage data,selecting, from among a plurality of reference fingerprints, a subset ofreference fingerprints; sending to the client device, the selectedsubset of reference fingerprints; receiving from the client device, anindication that the client device has detected a match between (i) oneof more of the selected subset of reference fingerprints, and (ii) oneor more fingerprints generated by the client device; and determining,based on the received indication from the client device, an identity ofa media sequence being presented on the client device.
 14. The serverdevice of claim 13, wherein the usage data indicates a state change of acontent selector module of the client device.
 15. The server device ofclaim 13, wherein the usage data indicates information about one or morereplacement media sequences sent to the client device.
 16. The serverdevice of claim 13, wherein the usage data indicates that the clientdevice is routinely tuned to a subset of multiple channels from among aplurality of channels, and wherein selecting the subset of referencefingerprints comprises selecting a subset of reference fingerprints thatcorresponds to the subset of multiple channels.
 17. The server device ofclaim 13, wherein selecting the subset of reference fingerprintscomprises: selecting, based on the received usage data, a subset ofmultiple channels from among a plurality of channels, each of theplurality of channels having corresponding reference fingerprints, andselecting the subset of reference fingerprints that correspond to theselected subset of multiple channels.
 18. The server device of claim 13,wherein the client device is a television and the media contentsequences are video content sequences.
 19. The server device of claim13, wherein the operations further comprise performing the recitedoperations in connection with each of multiple different client devicesand wherein a different subset of reference fingerprints is sent to eachof the multiple client devices.
 20. The server device of claim 13,wherein the operations further comprise: receiving from the clientdevice, a request for a replacement media content sequence, wherein therequest was generated by the client device in response to the clientdevice detecting the match between (i) the one or more referencefingerprints in the subset of reference fingerprints and (ii) the one ormore fingerprints generated by the client device; responsive toreceiving the request from the client device, selecting a replacementmedia content sequence; and performing an action that facilitatescausing the client device to present the selected replacement mediacontent sequence in place of an original media content sequence.
 21. Theserver device of claim 20, wherein the request for the replacement mediacontent sequence comprises the one or more fingerprints generated by theclient device.
 22. The server device of claim 20, wherein the requestfor the replacement media content sequence comprises an indication thatthe client device detected the match.
 23. The server device of claim 20,wherein the action comprises sending the selected replacement mediacontent sequence to the client device.
 24. The server device of claim20, wherein the original media content sequence is an originalcommercial and the replacement media content sequence is a replacementcommercial.
 25. A non-transitory computer-readable storage medium havingstored thereon program instructions that when executed by a serverdevice having one or more processors, cause the server device to performoperations comprising: receiving usage data for a client device that isconfigured to present media content sequences; based on the receivedusage data, selecting, from among a plurality of reference fingerprints,a subset of reference fingerprints; sending to the client device, theselected subset of reference fingerprints; receiving from the clientdevice, an indication that the client device has detected a matchbetween (i) one of more of the selected subset of referencefingerprints, and (ii) one or more fingerprints generated by the clientdevice; and determining, based on the received indication from theclient device, an identity of a media sequence being presented on theclient device.
 26. The non-transitory computer-readable storage mediumof claim 25, wherein the usage data indicates a state change of acontent selector module of the client device.
 27. The non-transitorycomputer-readable storage medium of claim 25, wherein the usage dataindicates information about one or more replacement media sequences sentto the client device.
 28. The non-transitory computer-readable storagemedium of claim 25, wherein the usage data indicates that the clientdevice is routinely tuned to a subset of multiple channels from among aplurality of channels, and wherein selecting the subset of referencefingerprints comprises selecting a subset of reference fingerprints thatcorresponds to the subset of multiple channels.
 29. The non-transitorycomputer-readable storage medium of claim 25, wherein selecting thesubset of reference fingerprints comprises: selecting, based on thereceived usage data, a subset of multiple channels from among aplurality of channels, each of the plurality of channels havingcorresponding reference fingerprints, and selecting the subset ofreference fingerprints that correspond to the selected subset ofmultiple channels.
 30. The non-transitory computer-readable storagemedium of claim 25, wherein the client device is a television and themedia content sequences are video content sequences.
 31. Thenon-transitory computer-readable storage medium of claim 25, wherein theoperations further comprise performing the recited operations inconnection with each of multiple different client devices and wherein adifferent subset of reference fingerprints is sent to each of themultiple client devices.
 32. The non-transitory computer-readablestorage medium of claim 25, wherein the operations further comprise:receiving from the client device, a request for a replacement mediacontent sequence, wherein the request was generated by the client devicein response to the client device detecting the match between (i) the oneor more reference fingerprints in the subset of reference fingerprintsand (ii) the one or more fingerprints generated by the client device;responsive to receiving the request from the client device, selecting areplacement media content sequence; and performing an action thatfacilitates causing the client device to present the selectedreplacement media content sequence in place of an original media contentsequence.
 33. The non-transitory computer-readable storage medium ofclaim 32, wherein the request for the replacement media content sequencecomprises the one or more fingerprints generated by the client device.34. The non-transitory computer-readable storage medium of claim 32,wherein the request for the replacement media content sequence comprisesan indication that the client device detected the match.
 35. Thenon-transitory computer-readable storage medium of claim 32, wherein theaction comprises sending the selected replacement media content sequenceto the client device.
 36. The non-transitory computer-readable storagemedium of claim 32, wherein the original media content sequence is anoriginal commercial and the replacement media content sequence is areplacement commercial.